[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:55:30 dist-test-slave-gkks krb5kdc[22642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:55:30 dist-test-slave-gkks krb5kdc[22642](info): set up 2 sockets
Sep 27 13:55:30 dist-test-slave-gkks krb5kdc[22642](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 13:55:31.984189 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:55:32 dist-test-slave-gkks krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:55:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:55:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:55:32.156303 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:34691
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:43111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:32.585211 22655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:32.586540 22655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:32.586745 22655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:32.594177 22655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:32.594292 22655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:32.594406 22655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:32.594517 22655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:32.594615 22655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:32.604646 22655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:43111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:32.609443 22655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:32.612401 22655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:32.638324 22660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:32.638348 22663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:32.638299 22661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:32.639751 22655 server_base.cc:1034] running on GCE node
I20260927 13:55:32.642264 22655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:32.643915 22655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:32.645161 22655 hybrid_clock.cc:648] HybridClock initialized: now 1790517332645105 us; error 69 us; skew 500 ppm
I20260927 13:55:32.646214 22655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:32 dist-test-slave-gkks krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517332, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:32.649926 22655 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:55:32.652172 22655 webserver.cc:460] Webserver started at http://127.22.26.254:33245/ using document root <none> and password file <none>
I20260927 13:55:32.654126 22655 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:32.654277 22655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:32.654631 22655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:32.661291 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "d57864dde39b46a2b74d3c2b2f23def2"
format_stamp: "Formatted at 2026-09-27 13:55:32 on dist-test-slave-gkks"
server_key: "1a573494a7db35227a4557b2314c54db"
server_key_iv: "bf1226a640d8f48421c255cb17030cba"
server_key_version: "encryptionkey@0"
I20260927 13:55:32.662122 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "d57864dde39b46a2b74d3c2b2f23def2"
format_stamp: "Formatted at 2026-09-27 13:55:32 on dist-test-slave-gkks"
server_key: "1a573494a7db35227a4557b2314c54db"
server_key_iv: "bf1226a640d8f48421c255cb17030cba"
server_key_version: "encryptionkey@0"
I20260927 13:55:32.669910 22655 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 13:55:32.679512 22670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:32.681020 22655 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 13:55:32.681239 22655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "d57864dde39b46a2b74d3c2b2f23def2"
format_stamp: "Formatted at 2026-09-27 13:55:32 on dist-test-slave-gkks"
server_key: "1a573494a7db35227a4557b2314c54db"
server_key_iv: "bf1226a640d8f48421c255cb17030cba"
server_key_version: "encryptionkey@0"
I20260927 13:55:32.681469 22655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:32.715773 22655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:32.720361 22655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:32.720666 22655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:32.743687 22655 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:34691
I20260927 13:55:32.743726 22721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:34691 every 8 connection(s)
I20260927 13:55:32.745743 22655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:55:32.754631 22722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:32.756143 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 22655
I20260927 13:55:32.756450 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:55:32.758292 22635 external_mini_cluster.cc:1428] Setting key 307d1ebe8df11f08506f7d981b667ef1
I20260927 13:55:32.774963 22722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2: Bootstrap starting.
I20260927 13:55:32.779863 22722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:32.781826 22722 log.cc:824] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:32.785089 22722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2: No bootstrap required, opened a new log
Sep 27 13:55:32 dist-test-slave-gkks krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:32.797955 22722 raft_consensus.cc:348] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57864dde39b46a2b74d3c2b2f23def2" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 34691 } }
I20260927 13:55:32.798393 22722 raft_consensus.cc:374] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:32.798511 22722 raft_consensus.cc:729] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57864dde39b46a2b74d3c2b2f23def2, State: Initialized, Role: FOLLOWER
I20260927 13:55:32.799858 22722 consensus_queue.cc:260] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57864dde39b46a2b74d3c2b2f23def2" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 34691 } }
I20260927 13:55:32.800099 22722 raft_consensus.cc:388] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:55:32.800246 22722 raft_consensus.cc:482] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:55:32.801251 22722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:32.802614 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:32.765054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51148 (local address 127.22.26.254:34691)
0927 13:55:32.766142 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:32.766155 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:32.767433 (+ 1278us) server_negotiation.cc:407] Connection header received
0927 13:55:32.776232 (+ 8799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:32.776253 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:32.776779 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:32.777204 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:32.780795 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:32.784108 (+ 3313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:32.786887 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:32.787781 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:32.788003 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:32.793707 (+ 5704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:32.793729 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:32.793745 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:32.793787 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:32.797005 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:32.797838 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:32.797844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:32.797850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:32.798006 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:32.798535 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:32.798544 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:32.798546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:32.799155 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:32.799480 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:32.800896 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0927 13:55:32.801343 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":627,"threads_started":1}
I20260927 13:55:32.804353 22722 raft_consensus.cc:504] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57864dde39b46a2b74d3c2b2f23def2" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 34691 } }
I20260927 13:55:32.805555 22722 leader_election.cc:302] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d57864dde39b46a2b74d3c2b2f23def2; no voters:
I20260927 13:55:32.806191 22722 leader_election.cc:288] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:55:32.806294 22727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:32.807116 22727 raft_consensus.cc:686] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [term 1 LEADER]: Becoming Leader. State: Replica: d57864dde39b46a2b74d3c2b2f23def2, State: Running, Role: LEADER
I20260927 13:55:32.807677 22727 consensus_queue.cc:237] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57864dde39b46a2b74d3c2b2f23def2" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 34691 } }
I20260927 13:55:32.810057 22722 sys_catalog.cc:564] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:32.823873 22728 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d57864dde39b46a2b74d3c2b2f23def2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57864dde39b46a2b74d3c2b2f23def2" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 34691 } } }
I20260927 13:55:32.823830 22729 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d57864dde39b46a2b74d3c2b2f23def2. Latest consensus state: current_term: 1 leader_uuid: "d57864dde39b46a2b74d3c2b2f23def2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57864dde39b46a2b74d3c2b2f23def2" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 34691 } } }
I20260927 13:55:32.824256 22728 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:32.824302 22729 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2 [sys.catalog]: This master's current role is: LEADER
W20260927 13:55:32.826490 22742 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:55:32.826627 22742 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:55:32.826781 22744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:55:32.828855 22744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:55:32.839396 22744 catalog_manager.cc:1347] Generated new cluster ID: f564d0a22ff844d58e2f3502e13ab3cf
I20260927 13:55:32.839564 22744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:55:32.854028 22744 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:55:32.855886 22744 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:55:32.872298 22744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d57864dde39b46a2b74d3c2b2f23def2: Generated new TSK 0
I20260927 13:55:32.873312 22744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:55:32.934990 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:34691
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:43111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:33.287436 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:33.287801 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:33.287993 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:33.295429 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:33.295579 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:33.295753 22748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:55:33.306221 22748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:43111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:34691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:33.307633 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:33.308981 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:33.322126 22754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:33.322242 22756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:33.322867 22753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:33.328316 22748 server_base.cc:1034] running on GCE node
I20260927 13:55:33.328989 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:33.329975 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:33.331194 22748 hybrid_clock.cc:648] HybridClock initialized: now 1790517333331146 us; error 62 us; skew 500 ppm
I20260927 13:55:33.331532 22748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:33 dist-test-slave-gkks krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517333, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:33.334745 22748 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:55:33.336364 22748 webserver.cc:460] Webserver started at http://127.22.26.193:32845/ using document root <none> and password file <none>
I20260927 13:55:33.337090 22748 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:33.337244 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:33.337564 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:33.340590 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "b9573e7d9cf3458aaf7fc35a272252a4"
format_stamp: "Formatted at 2026-09-27 13:55:33 on dist-test-slave-gkks"
server_key: "42399ae0abd45624808466ddc7d892d4"
server_key_iv: "1984e7f781becb9486e9a14eab662d45"
server_key_version: "encryptionkey@0"
I20260927 13:55:33.341388 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "b9573e7d9cf3458aaf7fc35a272252a4"
format_stamp: "Formatted at 2026-09-27 13:55:33 on dist-test-slave-gkks"
server_key: "42399ae0abd45624808466ddc7d892d4"
server_key_iv: "1984e7f781becb9486e9a14eab662d45"
server_key_version: "encryptionkey@0"
I20260927 13:55:33.347007 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:55:33.350566 22763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:33.352064 22748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 13:55:33.352252 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "b9573e7d9cf3458aaf7fc35a272252a4"
format_stamp: "Formatted at 2026-09-27 13:55:33 on dist-test-slave-gkks"
server_key: "42399ae0abd45624808466ddc7d892d4"
server_key_iv: "1984e7f781becb9486e9a14eab662d45"
server_key_version: "encryptionkey@0"
I20260927 13:55:33.352473 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:33.371380 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:33.375466 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:33.375752 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:33.376695 22748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:55:33.378007 22748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:33.378110 22748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:33.378268 22748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:33.378340 22748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:33.411783 22748 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:32805
I20260927 13:55:33.411871 22875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:32805 every 8 connection(s)
I20260927 13:55:33.413070 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:55:33.413700 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 22748
I20260927 13:55:33.413952 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:55:33.414553 22635 external_mini_cluster.cc:1428] Setting key 6813b0ca81fe7c0eaaae4cf7edf2b8fe
Sep 27 13:55:33 dist-test-slave-gkks krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517333, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
I20260927 13:55:33.437619 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:33.417176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:50855 (local address 127.22.26.254:34691)
0927 13:55:33.417652 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:33.417659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:33.418761 (+ 1102us) server_negotiation.cc:407] Connection header received
0927 13:55:33.421667 (+ 2906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:33.421673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:33.421753 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:33.421879 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:33.423531 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:33.424786 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:33.427186 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:33.427772 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:33.427898 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:33.431775 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:33.431781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:33.431786 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:33.431827 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:33.434446 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:33.435203 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:33.435208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:33.435212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:33.435328 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:33.435787 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:33.435792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:33.435795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:33.436060 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:33.436226 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:33.437018 (+ 792us) server_negotiation.cc:299] Negotiation successful
0927 13:55:33.437254 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":237,"threads_started":1}
I20260927 13:55:33.438542 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:33.417766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:34691 (local address 127.22.26.193:50855)
0927 13:55:33.418576 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:33.418632 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:33.421355 (+ 2723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:33.422143 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:33.422152 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:33.422636 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:33.423314 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:33.423344 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:33.424973 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:33.424982 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:33.426953 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:33.426971 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:33.427983 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:33.427991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:33.428244 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:33.428996 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:33.429021 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:33.431551 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:33.434589 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:33.434598 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:33.434615 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:33.435036 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:33.435468 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:33.435473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:33.435477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:33.435660 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:33.436224 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:33.436232 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:33.436595 (+ 363us) client_negotiation.cc:769] Sending connection context
0927 13:55:33.436909 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 13:55:33.437416 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":300,"threads_started":1}
I20260927 13:55:33.440582 22876 heartbeater.cc:347] Connected to a master server at 127.22.26.254:34691
I20260927 13:55:33.440900 22876 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:33.441598 22876 heartbeater.cc:511] Master 127.22.26.254:34691 requested a full tablet report, sending...
I20260927 13:55:33.445082 22687 ts_manager.cc:194] Registered new tserver with Master: b9573e7d9cf3458aaf7fc35a272252a4 (127.22.26.193:32805)
I20260927 13:55:33.448000 22687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:50855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:55:33.458443 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:34691
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:43111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:33.778445 22882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:33.778707 22882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:33.778801 22882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:33.786072 22882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:33.786180 22882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:33.786358 22882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:55:33.796474 22882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:43111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:34691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:33.797821 22882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:33.799232 22882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:33.812366 22887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:33.812438 22888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:33.812537 22890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:33.813390 22882 server_base.cc:1034] running on GCE node
I20260927 13:55:33.814060 22882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:33.815241 22882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:33.816499 22882 hybrid_clock.cc:648] HybridClock initialized: now 1790517333816448 us; error 54 us; skew 500 ppm
I20260927 13:55:33.816857 22882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:33 dist-test-slave-gkks krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517333, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:33.820302 22882 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:55:33.821975 22882 webserver.cc:460] Webserver started at http://127.22.26.194:44929/ using document root <none> and password file <none>
I20260927 13:55:33.822678 22882 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:33.822804 22882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:33.823158 22882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:33.826452 22882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "02de5a9aab0948249c801a98c047f28f"
format_stamp: "Formatted at 2026-09-27 13:55:33 on dist-test-slave-gkks"
server_key: "03608334f3941b96053659d7ca1a642b"
server_key_iv: "b2bf80047dda01a4920ec0d4aba276a9"
server_key_version: "encryptionkey@0"
I20260927 13:55:33.827415 22882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "02de5a9aab0948249c801a98c047f28f"
format_stamp: "Formatted at 2026-09-27 13:55:33 on dist-test-slave-gkks"
server_key: "03608334f3941b96053659d7ca1a642b"
server_key_iv: "b2bf80047dda01a4920ec0d4aba276a9"
server_key_version: "encryptionkey@0"
I20260927 13:55:33.833365 22882 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260927 13:55:33.837548 22897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:33.839250 22882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:55:33.839478 22882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "02de5a9aab0948249c801a98c047f28f"
format_stamp: "Formatted at 2026-09-27 13:55:33 on dist-test-slave-gkks"
server_key: "03608334f3941b96053659d7ca1a642b"
server_key_iv: "b2bf80047dda01a4920ec0d4aba276a9"
server_key_version: "encryptionkey@0"
I20260927 13:55:33.839730 22882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:33.864272 22882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:33.868809 22882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:33.869062 22882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:33.869987 22882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:55:33.871345 22882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:33.871416 22882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:33.871497 22882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:33.871541 22882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:33.904884 22882 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:35671
I20260927 13:55:33.904929 23009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:35671 every 8 connection(s)
I20260927 13:55:33.906157 22882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:55:33.914391 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 22882
I20260927 13:55:33.914582 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:55:33.915031 22635 external_mini_cluster.cc:1428] Setting key 294aa91ed9be31bc2f1c73fde0304e01
Sep 27 13:55:33 dist-test-slave-gkks krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517333, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:33.931288 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:33.908962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:37263 (local address 127.22.26.254:34691)
0927 13:55:33.909150 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:33.909154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:33.910351 (+ 1197us) server_negotiation.cc:407] Connection header received
0927 13:55:33.912534 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:33.912541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:33.912648 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:33.912773 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:33.914368 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:33.915362 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:33.917401 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:33.917859 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:33.918010 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:33.922926 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:33.922933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:33.922938 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:33.922977 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:33.927680 (+ 4703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:33.928722 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:33.928729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:33.928734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:33.928891 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:33.929427 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:33.929431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:33.929435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:33.929771 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:33.929927 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:33.930664 (+ 737us) server_negotiation.cc:299] Negotiation successful
0927 13:55:33.930914 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 13:55:33.932720 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:33.909328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:34691 (local address 127.22.26.194:37263)
0927 13:55:33.910164 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:33.910212 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:33.912241 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:33.913043 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:33.913053 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:33.913603 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:33.914183 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:33.914201 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:33.915525 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:33.915539 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:33.917240 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:33.917254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:33.918099 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:33.918106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:33.918362 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:33.919194 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:33.919217 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:33.922701 (+ 3484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:33.927852 (+ 5151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:33.927865 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:33.927885 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:33.928582 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:33.929050 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:33.929056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:33.929061 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:33.929277 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:33.929908 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:33.929916 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:33.930320 (+ 404us) client_negotiation.cc:769] Sending connection context
0927 13:55:33.930587 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 13:55:33.931157 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":334,"threads_started":1}
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
I20260927 13:55:33.934173 23010 heartbeater.cc:347] Connected to a master server at 127.22.26.254:34691
I20260927 13:55:33.934491 23010 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:33.935464 23010 heartbeater.cc:511] Master 127.22.26.254:34691 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
I20260927 13:55:33.937216 22687 ts_manager.cc:194] Registered new tserver with Master: 02de5a9aab0948249c801a98c047f28f (127.22.26.194:35671)
I20260927 13:55:33.938489 22687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:37263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:55:33.958989 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:34691
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:43111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:34.273381 23015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:34.273643 23015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:34.273813 23015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:34.280985 23015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:34.281095 23015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:34.281270 23015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:55:34.291154 23015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:43111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:34691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:34.292474 23015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:34.293834 23015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:34.306798 23020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:34.306723 23021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:34.306762 23023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:34.307668 23015 server_base.cc:1034] running on GCE node
I20260927 13:55:34.308297 23015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:34.309281 23015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:34.310519 23015 hybrid_clock.cc:648] HybridClock initialized: now 1790517334310481 us; error 54 us; skew 500 ppm
I20260927 13:55:34.310930 23015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:34 dist-test-slave-gkks krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517334, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:34.314188 23015 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:55:34.315713 23015 webserver.cc:460] Webserver started at http://127.22.26.195:39589/ using document root <none> and password file <none>
I20260927 13:55:34.316370 23015 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:34.316469 23015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:34.316721 23015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:34.319737 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "bf8a73caf76b4c95aae50c73e437f69d"
format_stamp: "Formatted at 2026-09-27 13:55:34 on dist-test-slave-gkks"
server_key: "560401bad358f88094775c0a18fd26f6"
server_key_iv: "0583617b6a6dd388a90487f5aad9445c"
server_key_version: "encryptionkey@0"
I20260927 13:55:34.320444 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "bf8a73caf76b4c95aae50c73e437f69d"
format_stamp: "Formatted at 2026-09-27 13:55:34 on dist-test-slave-gkks"
server_key: "560401bad358f88094775c0a18fd26f6"
server_key_iv: "0583617b6a6dd388a90487f5aad9445c"
server_key_version: "encryptionkey@0"
I20260927 13:55:34.325525 23015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:55:34.329034 23030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:34.330399 23015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:55:34.330564 23015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "bf8a73caf76b4c95aae50c73e437f69d"
format_stamp: "Formatted at 2026-09-27 13:55:34 on dist-test-slave-gkks"
server_key: "560401bad358f88094775c0a18fd26f6"
server_key_iv: "0583617b6a6dd388a90487f5aad9445c"
server_key_version: "encryptionkey@0"
I20260927 13:55:34.330749 23015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:34.343988 23015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:34.348278 23015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:34.348592 23015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:34.349539 23015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:55:34.350848 23015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:34.350921 23015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:34.351070 23015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:34.351207 23015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 13:55:34.385143 23015 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:45259
I20260927 13:55:34.385237 23142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:45259 every 8 connection(s)
I20260927 13:55:34.386543 23015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:55:34.394909 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 23015
I20260927 13:55:34.395191 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:55:34.395637 22635 external_mini_cluster.cc:1428] Setting key 7c2e2b90f972d2aabe5d762032d70cdc
Sep 27 13:55:34 dist-test-slave-gkks krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517334, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:34.407032 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:34.389713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:45325 (local address 127.22.26.254:34691)
0927 13:55:34.389891 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:34.389896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:34.390944 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 13:55:34.392867 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:34.392874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:34.392963 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:34.393072 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:34.394558 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.395433 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.397295 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.397947 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.398087 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:34.401797 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:34.401802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:34.401805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:34.401837 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:34.404405 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:34.405064 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:34.405068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:34.405071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:34.405190 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:34.405561 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:34.405564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:34.405566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:34.405821 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:34.405927 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:34.406565 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 13:55:34.406773 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:55:34.408026 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:34.390104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:34691 (local address 127.22.26.195:45325)
0927 13:55:34.390783 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:34.390828 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:34.392559 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:34.393320 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:34.393329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:34.393825 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:34.394389 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:34.394407 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.395583 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.395587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:34.397143 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:34.397156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.398158 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.398163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:34.398464 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:34.399322 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:34.399355 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:34.401568 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:34.404539 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:34.404546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:34.404559 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:34.404964 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:34.405279 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:34.405282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:34.405284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:34.405463 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:34.405919 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:34.405925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:34.406217 (+ 292us) client_negotiation.cc:769] Sending connection context
0927 13:55:34.406455 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 13:55:34.406828 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":259,"threads_started":1}
I20260927 13:55:34.409461 23143 heartbeater.cc:347] Connected to a master server at 127.22.26.254:34691
I20260927 13:55:34.409819 23143 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:34.410763 23143 heartbeater.cc:511] Master 127.22.26.254:34691 requested a full tablet report, sending...
I20260927 13:55:34.412648 22687 ts_manager.cc:194] Registered new tserver with Master: bf8a73caf76b4c95aae50c73e437f69d (127.22.26.195:45259)
I20260927 13:55:34.413547 22687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:45325
I20260927 13:55:34.424880 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 13:55:34 dist-test-slave-gkks krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517334, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260927 13:55:34.451218 22876 heartbeater.cc:503] Master 127.22.26.254:34691 was elected leader, sending a full tablet report...
Sep 27 13:55:34 dist-test-slave-gkks krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517334, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:55:34.469745 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:34.454814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.22.26.193:32805)
0927 13:55:34.455484 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:34.455491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:34.455584 (+ 93us) server_negotiation.cc:407] Connection header received
0927 13:55:34.455744 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:34.455750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:34.455959 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:34.456085 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:34.457545 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.458556 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.459813 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.460356 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.460467 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:34.464268 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:34.464278 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:34.464292 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:34.464321 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:34.467160 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:34.467776 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:34.467782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:34.467784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:34.467899 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:34.468251 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:34.468254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:34.468256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:34.468581 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:34.468823 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:34.469140 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 13:55:34.469366 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":386,"threads_started":1}
W20260927 13:55:34.471025 22791 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33540
Sep 27 13:55:34 dist-test-slave-gkks krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:55:34 dist-test-slave-gkks krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:55:34.515321 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:34.501276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.22.26.193:32805)
0927 13:55:34.501512 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:34.501518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:34.501714 (+ 196us) server_negotiation.cc:407] Connection header received
0927 13:55:34.501948 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:34.501952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:34.502047 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:34.502156 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:34.503565 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.504258 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.505485 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.505933 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.506073 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:34.510012 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:34.510017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:34.510021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:34.510055 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:34.512678 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:34.513411 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:34.513422 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:34.513424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:34.513541 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:34.513911 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:34.513916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:34.513920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:34.514177 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:34.514442 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:34.514735 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 13:55:34.514966 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 13:55:34.516887 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 22748
I20260927 13:55:34.529763 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:34.518585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.22.26.193:32805)
0927 13:55:34.518730 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:34.518734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:34.519180 (+ 446us) server_negotiation.cc:407] Connection header received
0927 13:55:34.519338 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:34.519342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:34.519397 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:34.519474 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:34.520645 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.521338 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.522597 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.522976 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.523063 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:34.525174 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:34.525178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:34.525181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:34.525207 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:34.527519 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:34.528081 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:34.528085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:34.528087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:34.528216 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:34.528659 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:34.528663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:34.528665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:34.528912 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:34.529092 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:34.529323 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 13:55:34.529467 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 13:55:34.530476 22871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:34.613385 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 22748
I20260927 13:55:34.630389 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 22882
Sep 27 13:55:34 dist-test-slave-gkks krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:55:34.646476 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:34.631376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.22.26.194:35671)
0927 13:55:34.631928 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:34.631936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:34.631960 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:55:34.632018 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:34.632025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:34.632232 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:34.632363 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:34.633530 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.634558 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.635946 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.636639 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.636848 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:34.641076 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:34.641086 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:34.641100 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:34.641148 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:34.643879 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:34.644613 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:34.644620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:34.644623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:34.644749 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:34.645123 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:34.645131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:34.645133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:34.645479 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:34.645705 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:34.645986 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 13:55:34.646198 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":293,"threads_started":1}
I20260927 13:55:34.647866 23005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:34.730736 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 22882
I20260927 13:55:34.749728 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 23015
Sep 27 13:55:34 dist-test-slave-gkks krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:55:34.765741 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:34.750728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.22.26.195:45259)
0927 13:55:34.751429 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:34.751435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:34.751465 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:55:34.751526 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:34.751532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:34.751747 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:34.751878 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:34.753085 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.754114 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.755515 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:34.756065 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:34.756160 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:34.760300 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:34.760311 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:34.760325 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:34.760373 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:34.763255 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:34.763911 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:34.763918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:34.763920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:34.764050 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:34.764406 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:34.764409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:34.764411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:34.764748 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:34.764977 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:34.765250 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 13:55:34.765460 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":370,"threads_started":1}
I20260927 13:55:34.767004 23138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:34.850116 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 23015
I20260927 13:55:34.866331 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 22655
I20260927 13:55:34.867610 22717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:34.971304 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 22655
2026-09-27T13:55:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4125 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:55:35 dist-test-slave-gkks krb5kdc[23164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:55:35 dist-test-slave-gkks krb5kdc[23164](info): set up 2 sockets
Sep 27 13:55:35 dist-test-slave-gkks krb5kdc[23164](info): commencing operation
krb5kdc: starting...
W20260927 13:55:36.055857 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:55:36 dist-test-slave-gkks krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:55:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:55:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:55:36.232381 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:35777
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:35777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:36.552635 23177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:36.552920 23177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:36.553007 23177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:36.560087 23177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:36.560187 23177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:36.560231 23177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:36.560297 23177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:36.560341 23177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:36.570443 23177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:35777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:35777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:36.571781 23177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:36.572985 23177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:36.584841 23182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:36.584853 23183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:36.584909 23185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:36.585856 23177 server_base.cc:1034] running on GCE node
I20260927 13:55:36.586613 23177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:36.588109 23177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:36.589330 23177 hybrid_clock.cc:648] HybridClock initialized: now 1790517336589288 us; error 36 us; skew 500 ppm
I20260927 13:55:36.589722 23177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:36 dist-test-slave-gkks krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517336, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:36.599691 23177 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:55:36.601316 23177 webserver.cc:460] Webserver started at http://127.22.26.254:44875/ using document root <none> and password file <none>
I20260927 13:55:36.602012 23177 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:36.602161 23177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:36.602483 23177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:36.605494 23177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "e39c59c61dac45cc817f17cf4487c2c7"
format_stamp: "Formatted at 2026-09-27 13:55:36 on dist-test-slave-gkks"
server_key: "5a07c382d0a4062ff24125313442761e"
server_key_iv: "1ef84acb49ec7d48a00c698fc84350f9"
server_key_version: "encryptionkey@0"
I20260927 13:55:36.606271 23177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "e39c59c61dac45cc817f17cf4487c2c7"
format_stamp: "Formatted at 2026-09-27 13:55:36 on dist-test-slave-gkks"
server_key: "5a07c382d0a4062ff24125313442761e"
server_key_iv: "1ef84acb49ec7d48a00c698fc84350f9"
server_key_version: "encryptionkey@0"
I20260927 13:55:36.611313 23177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260927 13:55:36.614609 23192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:36.616001 23177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:55:36.616179 23177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "e39c59c61dac45cc817f17cf4487c2c7"
format_stamp: "Formatted at 2026-09-27 13:55:36 on dist-test-slave-gkks"
server_key: "5a07c382d0a4062ff24125313442761e"
server_key_iv: "1ef84acb49ec7d48a00c698fc84350f9"
server_key_version: "encryptionkey@0"
I20260927 13:55:36.616370 23177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:36.657579 23177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:36.662201 23177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:36.662503 23177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:36.681100 23177 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:35777
I20260927 13:55:36.681123 23243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:35777 every 8 connection(s)
I20260927 13:55:36.682530 23177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:55:36.686179 23244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:36.689193 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 23177
I20260927 13:55:36.689437 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:55:36.690306 22635 external_mini_cluster.cc:1428] Setting key 702de9a8fa8e2c05d86b0f1b1e685c34
I20260927 13:55:36.698962 23244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7: Bootstrap starting.
I20260927 13:55:36.702729 23244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:36.704128 23244 log.cc:824] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7: Log is configured to *not* fsync() on all Append() calls
Sep 27 13:55:36 dist-test-slave-gkks krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:36.707408 23244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7: No bootstrap required, opened a new log
I20260927 13:55:36.714979 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:36.692418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.22.26.254:35777)
0927 13:55:36.693141 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:36.693153 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:36.693223 (+ 70us) server_negotiation.cc:407] Connection header received
0927 13:55:36.694954 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:36.694977 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:36.695466 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:36.695779 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:36.697594 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:36.698908 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:36.700689 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:36.701362 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:36.701491 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:36.706441 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:36.706452 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:36.706473 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:36.706529 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:36.710498 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:36.711177 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:36.711185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:36.711193 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:36.711407 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:36.711748 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:36.711753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:36.711757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:36.712456 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:36.712751 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:36.713170 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 13:55:36.713650 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":342,"threads_started":1}
I20260927 13:55:36.718029 23244 raft_consensus.cc:348] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39c59c61dac45cc817f17cf4487c2c7" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35777 } }
I20260927 13:55:36.718346 23244 raft_consensus.cc:374] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:36.718422 23244 raft_consensus.cc:729] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39c59c61dac45cc817f17cf4487c2c7, State: Initialized, Role: FOLLOWER
I20260927 13:55:36.718870 23244 consensus_queue.cc:260] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39c59c61dac45cc817f17cf4487c2c7" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35777 } }
I20260927 13:55:36.719158 23244 raft_consensus.cc:388] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:55:36.719311 23244 raft_consensus.cc:482] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:55:36.719485 23244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:36.722534 23244 raft_consensus.cc:504] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39c59c61dac45cc817f17cf4487c2c7" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35777 } }
I20260927 13:55:36.723006 23244 leader_election.cc:302] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e39c59c61dac45cc817f17cf4487c2c7; no voters:
I20260927 13:55:36.723631 23244 leader_election.cc:288] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:55:36.723762 23249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:36.724443 23249 raft_consensus.cc:686] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [term 1 LEADER]: Becoming Leader. State: Replica: e39c59c61dac45cc817f17cf4487c2c7, State: Running, Role: LEADER
I20260927 13:55:36.724982 23249 consensus_queue.cc:237] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39c59c61dac45cc817f17cf4487c2c7" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35777 } }
I20260927 13:55:36.725661 23244 sys_catalog.cc:564] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:36.729008 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e39c59c61dac45cc817f17cf4487c2c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39c59c61dac45cc817f17cf4487c2c7" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35777 } } }
I20260927 13:55:36.729034 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e39c59c61dac45cc817f17cf4487c2c7. Latest consensus state: current_term: 1 leader_uuid: "e39c59c61dac45cc817f17cf4487c2c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39c59c61dac45cc817f17cf4487c2c7" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35777 } } }
I20260927 13:55:36.729517 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:36.729517 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:36.730638 23259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:55:36.737147 23259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:55:36.745894 23259 catalog_manager.cc:1347] Generated new cluster ID: 75327aa06070433797baedb54c173ad8
I20260927 13:55:36.746009 23259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:55:36.759122 23259 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:55:36.760910 23259 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:55:36.777508 23259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e39c59c61dac45cc817f17cf4487c2c7: Generated new TSK 0
I20260927 13:55:36.778577 23259 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:55:36.832989 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35777
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:37.175354 23270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:37.175655 23270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:37.175837 23270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:37.183264 23270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:37.183401 23270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:37.183583 23270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:55:37.193202 23270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35777
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:37.194550 23270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:37.196072 23270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:37.209192 23275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:37.209228 23278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:37.209271 23276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:37.210282 23270 server_base.cc:1034] running on GCE node
I20260927 13:55:37.210881 23270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:37.211848 23270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:37.213044 23270 hybrid_clock.cc:648] HybridClock initialized: now 1790517337212991 us; error 45 us; skew 500 ppm
I20260927 13:55:37.213402 23270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:37 dist-test-slave-gkks krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517337, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:37.216845 23270 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:55:37.218261 23270 webserver.cc:460] Webserver started at http://127.22.26.193:33607/ using document root <none> and password file <none>
I20260927 13:55:37.218900 23270 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:37.219009 23270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:37.219339 23270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:37.222198 23270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "4effbe62b5e74c9aa46cbeefad451732"
format_stamp: "Formatted at 2026-09-27 13:55:37 on dist-test-slave-gkks"
server_key: "ffb95a64f42a8c34bba36b312b637d61"
server_key_iv: "425740f4691d346912666d04c691e66c"
server_key_version: "encryptionkey@0"
I20260927 13:55:37.222918 23270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "4effbe62b5e74c9aa46cbeefad451732"
format_stamp: "Formatted at 2026-09-27 13:55:37 on dist-test-slave-gkks"
server_key: "ffb95a64f42a8c34bba36b312b637d61"
server_key_iv: "425740f4691d346912666d04c691e66c"
server_key_version: "encryptionkey@0"
I20260927 13:55:37.228111 23270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 13:55:37.231809 23285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:37.233089 23270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:55:37.233210 23270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "4effbe62b5e74c9aa46cbeefad451732"
format_stamp: "Formatted at 2026-09-27 13:55:37 on dist-test-slave-gkks"
server_key: "ffb95a64f42a8c34bba36b312b637d61"
server_key_iv: "425740f4691d346912666d04c691e66c"
server_key_version: "encryptionkey@0"
I20260927 13:55:37.233367 23270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:37.250627 23270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:37.254860 23270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:37.255199 23270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:37.256132 23270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:55:37.257467 23270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:37.257539 23270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:37.257705 23270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:37.257772 23270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:37.292403 23270 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:38923
I20260927 13:55:37.292449 23397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:38923 every 8 connection(s)
I20260927 13:55:37.293682 23270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:55:37.302101 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 23270
I20260927 13:55:37.302327 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:55:37.302968 22635 external_mini_cluster.cc:1428] Setting key d593704ede00a61e9189411b0149574b
Sep 27 13:55:37 dist-test-slave-gkks krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517337, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:37.314695 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:37.296349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:38151 (local address 127.22.26.254:35777)
0927 13:55:37.296810 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:37.296814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:37.297609 (+ 795us) server_negotiation.cc:407] Connection header received
0927 13:55:37.299470 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:37.299476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:37.299567 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:37.299686 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:37.301137 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:37.301901 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:37.303774 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:37.304260 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:37.304365 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:37.308202 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:37.308212 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:37.308219 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:37.308268 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:37.311273 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:37.312231 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:37.312236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:37.312238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:37.312362 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:37.312873 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:37.312877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:37.312879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:37.313171 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:37.313279 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:37.314232 (+ 953us) server_negotiation.cc:299] Negotiation successful
0927 13:55:37.314411 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":362,"thread_start_us":239,"threads_started":1}
I20260927 13:55:37.316274 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:37.296778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35777 (local address 127.22.26.193:38151)
0927 13:55:37.297443 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:37.297486 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:37.299199 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:37.299921 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:37.299929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:37.300426 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:37.300961 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:37.300978 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:37.302065 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:37.302069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:37.303576 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:37.303594 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:37.304493 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:37.304500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:37.304847 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:37.305580 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:37.305600 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:37.307957 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:37.311411 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:37.311424 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:37.311444 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:37.312099 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:37.312472 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:37.312480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:37.312487 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:37.312733 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:37.313290 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:37.313302 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:37.313795 (+ 493us) client_negotiation.cc:769] Sending connection context
0927 13:55:37.314106 (+ 311us) client_negotiation.cc:240] Negotiation successful
0927 13:55:37.314685 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":255,"threads_started":1}
I20260927 13:55:37.318092 23398 heartbeater.cc:347] Connected to a master server at 127.22.26.254:35777
I20260927 13:55:37.318456 23398 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:37.319502 23398 heartbeater.cc:511] Master 127.22.26.254:35777 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
I20260927 13:55:37.322115 23209 ts_manager.cc:194] Registered new tserver with Master: 4effbe62b5e74c9aa46cbeefad451732 (127.22.26.193:38923)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
I20260927 13:55:37.323966 23209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:38151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:55:37.346369 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35777
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:37.663179 23404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:37.663504 23404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:37.663678 23404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:37.670913 23404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:37.671064 23404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:37.671291 23404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:55:37.680883 23404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35777
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:37.682112 23404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:37.683444 23404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:37.696297 23409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:37.696326 23410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:37.696848 23412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:37.697685 23404 server_base.cc:1034] running on GCE node
I20260927 13:55:37.698284 23404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:37.699221 23404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:37.700421 23404 hybrid_clock.cc:648] HybridClock initialized: now 1790517337700388 us; error 33 us; skew 500 ppm
I20260927 13:55:37.700731 23404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:37 dist-test-slave-gkks krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517337, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:37.703904 23404 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:55:37.705257 23404 webserver.cc:460] Webserver started at http://127.22.26.194:46033/ using document root <none> and password file <none>
I20260927 13:55:37.705927 23404 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:37.706073 23404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:37.706391 23404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:37.709240 23404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "420400db264d4ea29df5237751724331"
format_stamp: "Formatted at 2026-09-27 13:55:37 on dist-test-slave-gkks"
server_key: "8daa10432656967f1c66006766cefa79"
server_key_iv: "6062dcb71ff28d8858bf9ef0fd704b72"
server_key_version: "encryptionkey@0"
I20260927 13:55:37.710016 23404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "420400db264d4ea29df5237751724331"
format_stamp: "Formatted at 2026-09-27 13:55:37 on dist-test-slave-gkks"
server_key: "8daa10432656967f1c66006766cefa79"
server_key_iv: "6062dcb71ff28d8858bf9ef0fd704b72"
server_key_version: "encryptionkey@0"
I20260927 13:55:37.714897 23404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 13:55:37.718248 23419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:37.719619 23404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:55:37.719795 23404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "420400db264d4ea29df5237751724331"
format_stamp: "Formatted at 2026-09-27 13:55:37 on dist-test-slave-gkks"
server_key: "8daa10432656967f1c66006766cefa79"
server_key_iv: "6062dcb71ff28d8858bf9ef0fd704b72"
server_key_version: "encryptionkey@0"
I20260927 13:55:37.720069 23404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:37.730481 23404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:37.734820 23404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:37.735176 23404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:37.736101 23404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:55:37.737478 23404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:37.737579 23404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:37.737730 23404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:37.737804 23404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:37.771366 23404 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:46831
I20260927 13:55:37.771411 23531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:46831 every 8 connection(s)
I20260927 13:55:37.772670 23404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:55:37.782580 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 23404
I20260927 13:55:37.782837 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:55:37.783367 22635 external_mini_cluster.cc:1428] Setting key a7803a690c7cbc55364c2a4d4ce4d053
Sep 27 13:55:37 dist-test-slave-gkks krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517337, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:37.793375 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:37.775222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:43163 (local address 127.22.26.254:35777)
0927 13:55:37.775408 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:37.775413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:37.776580 (+ 1167us) server_negotiation.cc:407] Connection header received
0927 13:55:37.778599 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:37.778605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:37.778696 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:37.778809 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:37.780640 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:37.781384 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:37.783110 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:37.783603 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:37.783744 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:37.787666 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:37.787671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:37.787674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:37.787704 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:37.790278 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:37.791021 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:37.791030 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:37.791032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:37.791219 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:37.791674 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:37.791679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:37.791682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:37.791949 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:37.792117 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:37.792794 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 13:55:37.793023 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:55:37.794407 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:37.775746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35777 (local address 127.22.26.194:43163)
0927 13:55:37.776424 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:37.776463 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:37.778323 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:37.779043 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:37.779051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:37.779883 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:37.780449 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:37.780468 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:37.781548 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:37.781553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:37.782893 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:37.782908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:37.783831 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:37.783838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:37.784076 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:37.784883 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:37.784905 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:37.787434 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:37.790435 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:37.790444 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:37.790460 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:37.790888 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:37.791351 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:37.791356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:37.791359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:37.791549 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:37.792086 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:37.792095 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:37.792432 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 13:55:37.792696 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 13:55:37.793240 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":266,"threads_started":1}
I20260927 13:55:37.796083 23532 heartbeater.cc:347] Connected to a master server at 127.22.26.254:35777
I20260927 13:55:37.796469 23532 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:37.797497 23532 heartbeater.cc:511] Master 127.22.26.254:35777 requested a full tablet report, sending...
I20260927 13:55:37.799284 23209 ts_manager.cc:194] Registered new tserver with Master: 420400db264d4ea29df5237751724331 (127.22.26.194:46831)
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
I20260927 13:55:37.800599 23209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:43163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:55:37.825795 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35777
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:38.140890 23537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:38.141151 23537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:38.141309 23537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:38.148468 23537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:38.148563 23537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:38.148738 23537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:55:38.157955 23537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35777
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:38.159178 23537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:38.160481 23537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:38.173832 23542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:38.174281 23543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:38.174777 23545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:38.174810 23537 server_base.cc:1034] running on GCE node
I20260927 13:55:38.175509 23537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:38.176453 23537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:38.177682 23537 hybrid_clock.cc:648] HybridClock initialized: now 1790517338177627 us; error 69 us; skew 500 ppm
I20260927 13:55:38.178004 23537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:38 dist-test-slave-gkks krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517338, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:38.181278 23537 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:55:38.182837 23537 webserver.cc:460] Webserver started at http://127.22.26.195:34463/ using document root <none> and password file <none>
I20260927 13:55:38.183573 23537 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:38.183715 23537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:38.184026 23537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:38.186821 23537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "56a37f88916847ef94d833efb61186be"
format_stamp: "Formatted at 2026-09-27 13:55:38 on dist-test-slave-gkks"
server_key: "7a4254e48693d79e512425771e72c932"
server_key_iv: "71c11bc3c8c921dcf598e3b3840ad379"
server_key_version: "encryptionkey@0"
I20260927 13:55:38.187633 23537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "56a37f88916847ef94d833efb61186be"
format_stamp: "Formatted at 2026-09-27 13:55:38 on dist-test-slave-gkks"
server_key: "7a4254e48693d79e512425771e72c932"
server_key_iv: "71c11bc3c8c921dcf598e3b3840ad379"
server_key_version: "encryptionkey@0"
I20260927 13:55:38.192598 23537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:55:38.195911 23552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:38.197235 23537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:55:38.197422 23537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "56a37f88916847ef94d833efb61186be"
format_stamp: "Formatted at 2026-09-27 13:55:38 on dist-test-slave-gkks"
server_key: "7a4254e48693d79e512425771e72c932"
server_key_iv: "71c11bc3c8c921dcf598e3b3840ad379"
server_key_version: "encryptionkey@0"
I20260927 13:55:38.197613 23537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:38.216238 23537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:38.220531 23537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:38.220811 23537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:38.221719 23537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:55:38.222988 23537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:38.223088 23537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:38.223243 23537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:38.223325 23537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:38.256625 23537 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:44617
I20260927 13:55:38.256678 23664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:44617 every 8 connection(s)
I20260927 13:55:38.257896 23537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:55:38.261765 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 23537
I20260927 13:55:38.261984 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:55:38.262552 22635 external_mini_cluster.cc:1428] Setting key 50687eceacb9fdb47b0e0f5d3458e318
Sep 27 13:55:38 dist-test-slave-gkks krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517338, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:38.278924 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.260451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:49123 (local address 127.22.26.254:35777)
0927 13:55:38.260630 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:38.260635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:38.261904 (+ 1269us) server_negotiation.cc:407] Connection header received
0927 13:55:38.264146 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:38.264150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:38.264213 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:38.264291 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:38.266136 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.266912 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.269065 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.269547 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.269654 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.272986 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:38.272990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:38.272994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:38.273025 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:38.275798 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.276593 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.276598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.276601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.276755 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.277219 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.277223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.277225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.277563 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:38.277693 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:38.278525 (+ 832us) server_negotiation.cc:299] Negotiation successful
0927 13:55:38.278679 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:55:38.279860 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.260856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35777 (local address 127.22.26.195:49123)
0927 13:55:38.261668 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:38.261766 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:38.263889 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:38.264554 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:38.264562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:38.265256 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:38.265993 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:38.266012 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.267069 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.267074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:38.268913 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:38.268927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.269798 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.269804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:38.269992 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.270617 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:38.270672 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:38.272803 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:38.275920 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:38.275926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:38.275937 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:38.276493 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:38.276874 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:38.276877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:38.276879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:38.277102 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:38.277706 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:38.277714 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:38.278184 (+ 470us) client_negotiation.cc:769] Sending connection context
0927 13:55:38.278413 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 13:55:38.278792 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":291,"threads_started":1}
I20260927 13:55:38.281232 23665 heartbeater.cc:347] Connected to a master server at 127.22.26.254:35777
I20260927 13:55:38.281591 23665 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:38.282357 23665 heartbeater.cc:511] Master 127.22.26.254:35777 requested a full tablet report, sending...
I20260927 13:55:38.283953 23209 ts_manager.cc:194] Registered new tserver with Master: 56a37f88916847ef94d833efb61186be (127.22.26.195:44617)
I20260927 13:55:38.284783 23209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:49123
I20260927 13:55:38.290452 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:55:38.310665 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.298765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.22.26.254:35777)
0927 13:55:38.298936 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:38.298940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:38.299494 (+ 554us) server_negotiation.cc:407] Connection header received
0927 13:55:38.299627 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:38.299630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:38.299688 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:38.299771 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:38.301272 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.302056 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.303359 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.303730 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.303823 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.305894 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:38.305899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:38.305902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:38.305932 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:38.308548 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.309109 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.309113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.309115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.309229 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.309527 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.309530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.309532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.309813 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:38.309997 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:38.310285 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 13:55:38.310445 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:55:38.318136 23209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 13:55:38.327545 23398 heartbeater.cc:503] Master 127.22.26.254:35777 was elected leader, sending a full tablet report...
W20260927 13:55:38.331300 23209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:55:38.374233 23678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.350777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:38923 (local address 127.0.0.1:49076)
0927 13:55:38.364505 (+ 13728us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:38.364536 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:38.364689 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:38.366312 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:38.366317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:38.366338 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:38.367578 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:38.367587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.369646 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.369652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:38.371390 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:38.371402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.373225 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.373231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:38.373421 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.373478 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 13:55:38.373596 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 13:55:38.373783 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13548,"mutex_wait_us":80,"thread_start_us":13368,"threads_started":1}
I20260927 13:55:38.374423 23677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.350454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:44617 (local address 127.0.0.1:40252)
0927 13:55:38.357074 (+ 6620us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:38.357118 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:38.357306 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:38.366051 (+ 8745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:38.366058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:38.366088 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:38.367511 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:38.367535 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.369113 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.369122 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:38.371530 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:38.371546 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.373547 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.373556 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:38.373748 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.373783 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:55:38.373858 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 13:55:38.374010 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":448,"thread_start_us":1300,"threads_started":1}
I20260927 13:55:38.374724 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.351321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.22.26.193:38923)
0927 13:55:38.365704 (+ 14383us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:38.365712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:38.365739 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:55:38.365834 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:38.365842 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:38.366073 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:38.366954 (+ 881us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:38.367807 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.369475 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.371538 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.373006 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.373156 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.373381 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:38.373905 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 13:55:38.374074 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14225,"thread_start_us":14067,"threads_started":1}
I20260927 13:55:38.374817 23682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.351560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41550 (local address 127.22.26.194:46831)
0927 13:55:38.363712 (+ 12152us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:38.363725 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:38.366426 (+ 2701us) server_negotiation.cc:407] Connection header received
0927 13:55:38.366737 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:38.366745 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:38.366967 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:38.367168 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:38.368072 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.369599 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.372151 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.373458 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.373610 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.373781 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:38.374019 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 13:55:38.374188 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11929,"thread_start_us":12174,"threads_started":1}
I20260927 13:55:38.375286 23679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.350775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.22.26.195:44617)
0927 13:55:38.365338 (+ 14563us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:38.365348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:38.365386 (+ 38us) server_negotiation.cc:407] Connection header received
0927 13:55:38.365497 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:38.365507 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:38.365781 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:38.366544 (+ 763us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:38.367731 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.368973 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.371689 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.373043 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.373155 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.373383 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:38.374757 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0927 13:55:38.374925 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14377,"thread_start_us":14221,"threads_started":1}
I20260927 13:55:38.375525 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.351141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:46831 (local address 127.0.0.1:41550)
0927 13:55:38.366310 (+ 15169us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:38.366327 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:38.366530 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:38.367259 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:38.367263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:38.367287 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:38.367899 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:38.367913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.369742 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.369747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:38.371966 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:38.371983 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.373659 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.373664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:38.373802 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.373841 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 13:55:38.373902 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 13:55:38.375173 (+ 1271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15023,"mutex_wait_us":1167,"thread_start_us":14636,"threads_started":1}
I20260927 13:55:38.378100 23333 tablet_service.cc:1467] Processing CreateTablet for tablet 15ec43c4c46447cc99365e1fcfaadf62 (DEFAULT_TABLE table=test-table [id=0768f700f5424b35903ae95ab1b099dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:38.378140 23600 tablet_service.cc:1467] Processing CreateTablet for tablet 15ec43c4c46447cc99365e1fcfaadf62 (DEFAULT_TABLE table=test-table [id=0768f700f5424b35903ae95ab1b099dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:38.379463 23333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ec43c4c46447cc99365e1fcfaadf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:38.379488 23600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ec43c4c46447cc99365e1fcfaadf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:38.381951 23467 tablet_service.cc:1467] Processing CreateTablet for tablet 15ec43c4c46447cc99365e1fcfaadf62 (DEFAULT_TABLE table=test-table [id=0768f700f5424b35903ae95ab1b099dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:38.383231 23467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ec43c4c46447cc99365e1fcfaadf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:38.390457 23683 tablet_bootstrap.cc:492] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be: Bootstrap starting.
I20260927 13:55:38.393442 23684 tablet_bootstrap.cc:492] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732: Bootstrap starting.
I20260927 13:55:38.393846 23683 tablet_bootstrap.cc:654] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:38.395380 23683 log.cc:824] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:38.396458 23685 tablet_bootstrap.cc:492] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331: Bootstrap starting.
I20260927 13:55:38.397186 23684 tablet_bootstrap.cc:654] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:38.398473 23684 log.cc:824] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:38.399967 23683 tablet_bootstrap.cc:492] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be: No bootstrap required, opened a new log
I20260927 13:55:38.399976 23685 tablet_bootstrap.cc:654] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:38.400249 23683 ts_tablet_manager.cc:1397] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 13:55:38.401196 23684 tablet_bootstrap.cc:492] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732: No bootstrap required, opened a new log
I20260927 13:55:38.401190 23685 log.cc:824] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:38.401463 23684 ts_tablet_manager.cc:1397] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 13:55:38.404270 23685 tablet_bootstrap.cc:492] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331: No bootstrap required, opened a new log
I20260927 13:55:38.404521 23685 ts_tablet_manager.cc:1397] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 13:55:38.408895 23684 raft_consensus.cc:348] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a37f88916847ef94d833efb61186be" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44617 } } peers { permanent_uuid: "4effbe62b5e74c9aa46cbeefad451732" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38923 } } peers { permanent_uuid: "420400db264d4ea29df5237751724331" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 46831 } }
I20260927 13:55:38.409250 23684 raft_consensus.cc:374] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:38.409387 23684 raft_consensus.cc:729] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4effbe62b5e74c9aa46cbeefad451732, State: Initialized, Role: FOLLOWER
I20260927 13:55:38.409984 23684 consensus_queue.cc:260] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a37f88916847ef94d833efb61186be" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44617 } } peers { permanent_uuid: "4effbe62b5e74c9aa46cbeefad451732" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38923 } } peers { permanent_uuid: "420400db264d4ea29df5237751724331" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 46831 } }
I20260927 13:55:38.410759 23683 raft_consensus.cc:348] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a37f88916847ef94d833efb61186be" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44617 } } peers { permanent_uuid: "4effbe62b5e74c9aa46cbeefad451732" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38923 } } peers { permanent_uuid: "420400db264d4ea29df5237751724331" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 46831 } }
I20260927 13:55:38.411197 23683 raft_consensus.cc:374] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:38.411311 23683 raft_consensus.cc:729] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a37f88916847ef94d833efb61186be, State: Initialized, Role: FOLLOWER
I20260927 13:55:38.411758 23684 ts_tablet_manager.cc:1428] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260927 13:55:38.411875 23683 consensus_queue.cc:260] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a37f88916847ef94d833efb61186be" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44617 } } peers { permanent_uuid: "4effbe62b5e74c9aa46cbeefad451732" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38923 } } peers { permanent_uuid: "420400db264d4ea29df5237751724331" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 46831 } }
I20260927 13:55:38.413579 23683 ts_tablet_manager.cc:1428] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260927 13:55:38.413944 23690 raft_consensus.cc:482] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:55:38.414016 23665 heartbeater.cc:503] Master 127.22.26.254:35777 was elected leader, sending a full tablet report...
I20260927 13:55:38.414357 23690 raft_consensus.cc:504] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a37f88916847ef94d833efb61186be" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44617 } } peers { permanent_uuid: "4effbe62b5e74c9aa46cbeefad451732" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38923 } } peers { permanent_uuid: "420400db264d4ea29df5237751724331" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 46831 } }
I20260927 13:55:38.416231 23690 leader_election.cc:288] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4effbe62b5e74c9aa46cbeefad451732 (127.22.26.193:38923), 420400db264d4ea29df5237751724331 (127.22.26.194:46831)
I20260927 13:55:38.416678 23685 raft_consensus.cc:348] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a37f88916847ef94d833efb61186be" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44617 } } peers { permanent_uuid: "4effbe62b5e74c9aa46cbeefad451732" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38923 } } peers { permanent_uuid: "420400db264d4ea29df5237751724331" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 46831 } }
I20260927 13:55:38.419939 23685 raft_consensus.cc:374] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:38.420053 23685 raft_consensus.cc:729] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420400db264d4ea29df5237751724331, State: Initialized, Role: FOLLOWER
I20260927 13:55:38.420746 23685 consensus_queue.cc:260] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a37f88916847ef94d833efb61186be" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44617 } } peers { permanent_uuid: "4effbe62b5e74c9aa46cbeefad451732" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38923 } } peers { permanent_uuid: "420400db264d4ea29df5237751724331" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 46831 } }
I20260927 13:55:38.421861 23532 heartbeater.cc:503] Master 127.22.26.254:35777 was elected leader, sending a full tablet report...
I20260927 13:55:38.422433 23685 ts_tablet_manager.cc:1428] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
I20260927 13:55:38.424631 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.416708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:46831 (local address 127.22.26.195:42619)
0927 13:55:38.416907 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:38.416929 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:38.417059 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:38.417756 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:38.417760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:38.417785 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:38.418029 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:38.418045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.419768 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.419774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:38.421830 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:38.421844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.423711 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.423717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:38.424009 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.424059 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 13:55:38.424155 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 13:55:38.424299 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":28}
I20260927 13:55:38.425259 23682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.417146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:42619 (local address 127.22.26.194:46831)
0927 13:55:38.417321 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:38.417326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:38.417345 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:55:38.417413 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:38.417419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:38.417494 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:38.417585 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:38.418252 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.419580 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.422027 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.423372 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.423543 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.423615 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:38.424888 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0927 13:55:38.425007 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 13:55:38.426184 23487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ec43c4c46447cc99365e1fcfaadf62" candidate_uuid: "56a37f88916847ef94d833efb61186be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "420400db264d4ea29df5237751724331" is_pre_election: true
I20260927 13:55:38.426496 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.418012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:60173 (local address 127.22.26.193:38923)
0927 13:55:38.418631 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:38.418639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:38.419597 (+ 958us) server_negotiation.cc:407] Connection header received
0927 13:55:38.419869 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:38.419875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:38.419943 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:38.420005 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:38.420915 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.421996 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.424172 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.425322 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.425477 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.425545 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:38.426125 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 13:55:38.426273 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481}
I20260927 13:55:38.426594 23692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.417727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:38923 (local address 127.22.26.195:60173)
0927 13:55:38.419485 (+ 1758us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:38.419510 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:38.419695 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:38.420229 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:38.420234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:38.420257 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:38.420727 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:38.420742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.422183 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.422188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:38.424019 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:38.424031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.425720 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.425726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:38.425918 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.425957 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 13:55:38.426056 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 13:55:38.426192 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1626,"mutex_wait_us":1,"thread_start_us":1474,"threads_started":1}
I20260927 13:55:38.426692 23487 raft_consensus.cc:2454] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56a37f88916847ef94d833efb61186be in term 0.
I20260927 13:55:38.427493 23553 leader_election.cc:302] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 420400db264d4ea29df5237751724331, 56a37f88916847ef94d833efb61186be; no voters:
I20260927 13:55:38.428232 23353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ec43c4c46447cc99365e1fcfaadf62" candidate_uuid: "56a37f88916847ef94d833efb61186be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4effbe62b5e74c9aa46cbeefad451732" is_pre_election: true
I20260927 13:55:38.428694 23353 raft_consensus.cc:2454] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56a37f88916847ef94d833efb61186be in term 0.
I20260927 13:55:38.429152 23691 raft_consensus.cc:2781] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:55:38.429342 23691 raft_consensus.cc:482] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:55:38.429492 23691 raft_consensus.cc:3037] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:38.431734 23691 raft_consensus.cc:504] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a37f88916847ef94d833efb61186be" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44617 } } peers { permanent_uuid: "4effbe62b5e74c9aa46cbeefad451732" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38923 } } peers { permanent_uuid: "420400db264d4ea29df5237751724331" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 46831 } }
I20260927 13:55:38.432327 23691 leader_election.cc:288] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [CANDIDATE]: Term 1 election: Requested vote from peers 4effbe62b5e74c9aa46cbeefad451732 (127.22.26.193:38923), 420400db264d4ea29df5237751724331 (127.22.26.194:46831)
I20260927 13:55:38.432621 23353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ec43c4c46447cc99365e1fcfaadf62" candidate_uuid: "56a37f88916847ef94d833efb61186be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4effbe62b5e74c9aa46cbeefad451732"
I20260927 13:55:38.432858 23353 raft_consensus.cc:3037] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:38.432847 23487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ec43c4c46447cc99365e1fcfaadf62" candidate_uuid: "56a37f88916847ef94d833efb61186be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "420400db264d4ea29df5237751724331"
I20260927 13:55:38.433017 23487 raft_consensus.cc:3037] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:38.434985 23353 raft_consensus.cc:2454] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56a37f88916847ef94d833efb61186be in term 1.
I20260927 13:55:38.435151 23487 raft_consensus.cc:2454] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56a37f88916847ef94d833efb61186be in term 1.
I20260927 13:55:38.435438 23556 leader_election.cc:302] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4effbe62b5e74c9aa46cbeefad451732, 56a37f88916847ef94d833efb61186be; no voters:
I20260927 13:55:38.435693 23691 raft_consensus.cc:2781] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:38.435885 23691 raft_consensus.cc:686] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [term 1 LEADER]: Becoming Leader. State: Replica: 56a37f88916847ef94d833efb61186be, State: Running, Role: LEADER
I20260927 13:55:38.436303 23691 consensus_queue.cc:237] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a37f88916847ef94d833efb61186be" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44617 } } peers { permanent_uuid: "4effbe62b5e74c9aa46cbeefad451732" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38923 } } peers { permanent_uuid: "420400db264d4ea29df5237751724331" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 46831 } }
I20260927 13:55:38.443411 23209 catalog_manager.cc:5495] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be reported cstate change: term changed from 0 to 1, leader changed from <none> to 56a37f88916847ef94d833efb61186be (127.22.26.195). New cstate: current_term: 1 leader_uuid: "56a37f88916847ef94d833efb61186be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a37f88916847ef94d833efb61186be" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4effbe62b5e74c9aa46cbeefad451732" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "420400db264d4ea29df5237751724331" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 46831 } health_report { overall_health: UNKNOWN } } }
Sep 27 13:55:38 dist-test-slave-gkks krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:55:38.463348 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.449565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.22.26.193:38923)
0927 13:55:38.449720 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:38.449724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:38.450213 (+ 489us) server_negotiation.cc:407] Connection header received
0927 13:55:38.450353 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:38.450356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:38.450417 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:38.450497 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:38.451941 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.452649 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.453832 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.454282 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.454384 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.458110 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:38.458122 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:38.458136 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:38.458171 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:38.461044 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.461708 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.461715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.461717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.461846 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.462137 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.462140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.462142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.462470 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:38.462682 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:38.462887 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 13:55:38.463034 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 13:55:38 dist-test-slave-gkks krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517338, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:55:38 dist-test-slave-gkks krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517338, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:55:38.508996 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.495038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.22.26.193:38923)
0927 13:55:38.495291 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:38.495297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:38.495581 (+ 284us) server_negotiation.cc:407] Connection header received
0927 13:55:38.495794 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:38.495804 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:38.495868 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:38.495973 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:38.497415 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.498144 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.499504 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.499903 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.500038 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.503990 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:38.503995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:38.503998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:38.504030 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:38.506635 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.507302 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.507306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.507308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.507427 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.507833 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.507836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.507839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.508073 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:38.508255 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:38.508565 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 13:55:38.508725 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260927 13:55:38.509511 23666 tablet.cc:2369] T 15ec43c4c46447cc99365e1fcfaadf62 P 56a37f88916847ef94d833efb61186be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:55:38.524767 23533 tablet.cc:2369] T 15ec43c4c46447cc99365e1fcfaadf62 P 420400db264d4ea29df5237751724331: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 13:55:38 dist-test-slave-gkks krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260927 13:55:38.545948 23399 tablet.cc:2369] T 15ec43c4c46447cc99365e1fcfaadf62 P 4effbe62b5e74c9aa46cbeefad451732: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 13:55:38 dist-test-slave-gkks krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:55:38.554087 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.540482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.22.26.193:38923)
0927 13:55:38.540702 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:38.540710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:38.541041 (+ 331us) server_negotiation.cc:407] Connection header received
0927 13:55:38.541250 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:38.541261 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:38.541325 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:38.541434 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:38.542773 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.543531 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.544751 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.545167 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.545303 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.549460 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:38.549464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:38.549467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:38.549498 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:38.551901 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.552456 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.552459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.552462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.552574 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.552984 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.552988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.552990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.553257 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:38.553433 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:38.553671 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 13:55:38.553837 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 13:55:38.555824 23313 tablet_service.cc:2908] Rejecting scan request for tablet 15ec43c4c46447cc99365e1fcfaadf62: Uninitialized: clean time has not yet been initialized
I20260927 13:55:38.559696 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 23270
I20260927 13:55:38.571463 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.560588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.22.26.193:38923)
0927 13:55:38.560739 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:38.560743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:38.561167 (+ 424us) server_negotiation.cc:407] Connection header received
0927 13:55:38.561311 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:38.561315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:38.561369 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:38.561445 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:38.562518 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.563221 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.564322 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.564719 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.564816 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.566780 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:38.566784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:38.566787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:38.566815 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:38.569365 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.569970 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.569973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.569975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.570081 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.570381 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.570384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.570386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.570611 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:38.570843 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:38.571017 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 13:55:38.571240 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:55:38.572085 23393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:38.661607 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 23270
I20260927 13:55:38.678224 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 23404
Sep 27 13:55:38 dist-test-slave-gkks krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:55:38.693699 23682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.679060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.22.26.194:46831)
0927 13:55:38.679278 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:38.679285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:38.679371 (+ 86us) server_negotiation.cc:407] Connection header received
0927 13:55:38.679599 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:38.679609 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:38.679674 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:38.679781 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:38.681002 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.681761 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.683045 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.683545 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.683687 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.687895 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:38.687910 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:38.687926 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:38.687963 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:38.690759 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.691495 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.691502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.691505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.691631 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.692035 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.692039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.692042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.692555 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:38.692884 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:38.693164 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 13:55:38.693384 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 13:55:38.694365 23527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:38.782156 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 23404
I20260927 13:55:38.798785 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 23537
Sep 27 13:55:38 dist-test-slave-gkks krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:55:38.814173 23679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:38.799654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.22.26.195:44617)
0927 13:55:38.799839 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:38.799847 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:38.799902 (+ 55us) server_negotiation.cc:407] Connection header received
0927 13:55:38.800118 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:38.800123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:38.800200 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:38.800314 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:38.801636 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.802348 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.803878 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:38.804321 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:38.804461 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:38.808536 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:38.808551 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:38.808566 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:38.808601 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:38.811463 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.812117 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.812134 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.812136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.812273 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:38.812669 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:38.812675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:38.812678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:38.813027 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:38.813341 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:38.813607 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 13:55:38.813868 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:55:38.814860 23660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:38.903920 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 23537
I20260927 13:55:38.920392 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 23177
I20260927 13:55:38.921531 23239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:39.031821 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 23177
2026-09-27T13:55:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4061 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:55:39 dist-test-slave-gkks krb5kdc[23712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:55:39 dist-test-slave-gkks krb5kdc[23712](info): set up 2 sockets
Sep 27 13:55:39 dist-test-slave-gkks krb5kdc[23712](info): commencing operation
krb5kdc: starting...
W20260927 13:55:40.123683 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.007s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:55:40 dist-test-slave-gkks krb5kdc[23712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:55:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:55:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:55:40.299635 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:45575
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:45575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:40.612596 23725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:40.612897 23725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:40.613062 23725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:40.620306 23725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:40.620417 23725 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 13:55:40.620491 23725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:40.620604 23725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:40.620692 23725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:40.620810 23725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:40.630776 23725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:45575
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:45575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:40.632081 23725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:40.633396 23725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:40.645376 23730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:40.645326 23731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:40.645438 23733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:40.646600 23725 server_base.cc:1034] running on GCE node
I20260927 13:55:40.647428 23725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:40.648856 23725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:40.650121 23725 hybrid_clock.cc:648] HybridClock initialized: now 1790517340650061 us; error 45 us; skew 500 ppm
I20260927 13:55:40.650480 23725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:40 dist-test-slave-gkks krb5kdc[23712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517340, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:40.653998 23725 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:55:40.655613 23725 webserver.cc:460] Webserver started at http://127.22.26.254:45907/ using document root <none> and password file <none>
I20260927 13:55:40.656373 23725 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:40.656519 23725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:40.656859 23725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:40.660226 23725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "6304f134852d4b819ff319e59998e003"
format_stamp: "Formatted at 2026-09-27 13:55:40 on dist-test-slave-gkks"
server_key: "37dc6cd88200467f89252559c3f65a7c"
server_key_iv: "24af74f1c7a7710614a03324e41a8d34"
server_key_version: "encryptionkey@0"
I20260927 13:55:40.661357 23725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "6304f134852d4b819ff319e59998e003"
format_stamp: "Formatted at 2026-09-27 13:55:40 on dist-test-slave-gkks"
server_key: "37dc6cd88200467f89252559c3f65a7c"
server_key_iv: "24af74f1c7a7710614a03324e41a8d34"
server_key_version: "encryptionkey@0"
I20260927 13:55:40.667061 23725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 13:55:40.670990 23740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:40.672629 23725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 13:55:40.672860 23725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "6304f134852d4b819ff319e59998e003"
format_stamp: "Formatted at 2026-09-27 13:55:40 on dist-test-slave-gkks"
server_key: "37dc6cd88200467f89252559c3f65a7c"
server_key_iv: "24af74f1c7a7710614a03324e41a8d34"
server_key_version: "encryptionkey@0"
I20260927 13:55:40.673116 23725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:40.688215 23725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:40.692507 23725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:40.692797 23725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:40.715127 23725 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:45575
I20260927 13:55:40.715056 23801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:45575 every 8 connection(s)
I20260927 13:55:40.716585 23725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:55:40.720391 23802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:40.725471 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 23725
I20260927 13:55:40.725710 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:55:40.726348 22635 external_mini_cluster.cc:1428] Setting key 1df646f2a82a6c55a30f0f73e9dc7056
I20260927 13:55:40.730885 23802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003: Bootstrap starting.
I20260927 13:55:40.734496 23802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:40.735769 23802 log.cc:824] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:40.738876 23802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003: No bootstrap required, opened a new log
Sep 27 13:55:40 dist-test-slave-gkks krb5kdc[23712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:40.749346 23802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6304f134852d4b819ff319e59998e003" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 45575 } }
I20260927 13:55:40.749770 23802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:40.749876 23802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6304f134852d4b819ff319e59998e003, State: Initialized, Role: FOLLOWER
I20260927 13:55:40.750556 23802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6304f134852d4b819ff319e59998e003" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 45575 } }
I20260927 13:55:40.750845 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:40.728316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.22.26.254:45575)
0927 13:55:40.729161 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:40.729172 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:40.729209 (+ 37us) server_negotiation.cc:407] Connection header received
0927 13:55:40.730820 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:40.730840 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:40.731301 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:40.731597 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:40.733334 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:40.734675 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:40.736485 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:40.737345 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:40.737477 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:40.742137 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:40.742147 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:40.742165 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:40.742210 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:40.746342 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:40.746993 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:40.747004 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:40.747015 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:40.747267 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:40.747611 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:40.747622 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:40.747626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:40.748386 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:40.748656 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:40.749004 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 13:55:40.749428 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":418,"threads_started":1}
I20260927 13:55:40.750852 23802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:55:40.751046 23802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:55:40.751258 23802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:40.753666 23802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6304f134852d4b819ff319e59998e003" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 45575 } }
I20260927 13:55:40.754168 23802 leader_election.cc:302] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6304f134852d4b819ff319e59998e003; no voters:
I20260927 13:55:40.754762 23802 leader_election.cc:288] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:55:40.754895 23807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:40.755638 23807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [term 1 LEADER]: Becoming Leader. State: Replica: 6304f134852d4b819ff319e59998e003, State: Running, Role: LEADER
I20260927 13:55:40.756152 23807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6304f134852d4b819ff319e59998e003" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 45575 } }
I20260927 13:55:40.756894 23802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:40.759578 23808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6304f134852d4b819ff319e59998e003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6304f134852d4b819ff319e59998e003" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 45575 } } }
I20260927 13:55:40.759933 23808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:40.759680 23809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6304f134852d4b819ff319e59998e003. Latest consensus state: current_term: 1 leader_uuid: "6304f134852d4b819ff319e59998e003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6304f134852d4b819ff319e59998e003" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 45575 } } }
I20260927 13:55:40.760067 23809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:40.761843 23814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:55:40.768075 23814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:55:40.776357 23814 catalog_manager.cc:1347] Generated new cluster ID: 6f827b2819b34a07a5e27eadefef1921
I20260927 13:55:40.776494 23814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:55:40.800033 23814 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:55:40.801370 23814 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:55:40.809733 23814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003: Generated new TSK 0
I20260927 13:55:40.810652 23814 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:55:40.854956 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:45575
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:41.178745 23828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:41.179147 23828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:41.179370 23828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:41.189260 23828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:41.189440 23828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:41.189637 23828 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:55:41.189815 23828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:55:41.200961 23828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.26.254:45575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:41.202345 23828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:41.203899 23828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:41.217199 23833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:41.217247 23836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:41.217247 23834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:41.218148 23828 server_base.cc:1034] running on GCE node
I20260927 13:55:41.218811 23828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:41.219878 23828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:41.221125 23828 hybrid_clock.cc:648] HybridClock initialized: now 1790517341221060 us; error 26 us; skew 500 ppm
I20260927 13:55:41.221468 23828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:41 dist-test-slave-gkks krb5kdc[23712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517341, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:41.224673 23828 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:55:41.226106 23828 webserver.cc:460] Webserver started at http://127.22.26.193:46775/ using document root <none> and password file <none>
I20260927 13:55:41.227074 23828 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:41.227259 23828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:41.227639 23828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:41.230628 23828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "bbb135d70fdd4a1db14249aaf892d8fc"
format_stamp: "Formatted at 2026-09-27 13:55:41 on dist-test-slave-gkks"
server_key: "2915c6672b564fe51687cfaf1fc0c27f"
server_key_iv: "86c5e8bbefe62e79b7f41de0f3ccf572"
server_key_version: "encryptionkey@0"
I20260927 13:55:41.231520 23828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "bbb135d70fdd4a1db14249aaf892d8fc"
format_stamp: "Formatted at 2026-09-27 13:55:41 on dist-test-slave-gkks"
server_key: "2915c6672b564fe51687cfaf1fc0c27f"
server_key_iv: "86c5e8bbefe62e79b7f41de0f3ccf572"
server_key_version: "encryptionkey@0"
I20260927 13:55:41.236761 23828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:55:41.240212 23843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:41.241581 23828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:55:41.241760 23828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "bbb135d70fdd4a1db14249aaf892d8fc"
format_stamp: "Formatted at 2026-09-27 13:55:41 on dist-test-slave-gkks"
server_key: "2915c6672b564fe51687cfaf1fc0c27f"
server_key_iv: "86c5e8bbefe62e79b7f41de0f3ccf572"
server_key_version: "encryptionkey@0"
I20260927 13:55:41.241982 23828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:41.255074 23828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:41.259264 23828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:41.259541 23828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:41.263772 23828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:41.263882 23828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:41.264022 23828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:41.264104 23828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:55:41 dist-test-slave-gkks krb5kdc[23712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517341, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:41.296325 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:41.264186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:35965 (local address 127.22.26.254:45575)
0927 13:55:41.271599 (+ 7413us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:41.271603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:41.271620 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:55:41.271672 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:41.271676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:41.271734 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:41.271829 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:41.275692 (+ 3863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.277050 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.280044 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.280736 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.283703 (+ 2967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:41.285998 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:41.286007 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:41.286016 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:41.286068 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:41.290719 (+ 4651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:41.291966 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:41.291976 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:41.291981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:41.292242 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:41.293209 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:41.293218 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:41.293222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:41.293867 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:41.295239 (+ 1372us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:41.295449 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 13:55:41.295768 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7297,"thread_start_us":6999,"threads_started":1}
I20260927 13:55:41.297482 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:41.265161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:45575 (local address 127.22.26.193:35965)
0927 13:55:41.266209 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:41.266268 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:41.269611 (+ 3343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:41.273555 (+ 3944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:41.273569 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:41.274393 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:41.275305 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:41.275334 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.277280 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.277293 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:41.279821 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:41.279840 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.281038 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.281048 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:41.281371 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:41.282262 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:41.282299 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:41.285637 (+ 3338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:41.290926 (+ 5289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:41.290941 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:41.290962 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:41.291733 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:41.292445 (+ 712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:41.292450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:41.292454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:41.292942 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:41.294128 (+ 1186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:41.294176 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:41.294881 (+ 705us) client_negotiation.cc:769] Sending connection context
0927 13:55:41.295278 (+ 397us) client_negotiation.cc:240] Negotiation successful
0927 13:55:41.295945 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":524,"threads_started":1}
I20260927 13:55:41.325899 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:41.306291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:37661 (local address 127.22.26.254:45575)
0927 13:55:41.306478 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:41.306485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:41.311251 (+ 4766us) server_negotiation.cc:407] Connection header received
0927 13:55:41.311332 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:41.311337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:41.311416 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:41.311505 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:41.312673 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.313735 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.315570 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.316188 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.316318 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:41.318761 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:41.318768 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:41.318773 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:41.318822 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:41.322784 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:41.323824 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:41.323832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:41.323840 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:41.323993 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:41.324402 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:41.324407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:41.324410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:41.324797 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:41.324936 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:41.325360 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 13:55:41.325572 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 13:55:41.326009 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:41.306345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:45575 (local address 127.22.26.193:37661)
0927 13:55:41.307021 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:41.307042 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:41.307233 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:41.311677 (+ 4444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:41.311682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:41.312170 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:41.312496 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:41.312508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.313916 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.313921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:41.315413 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:41.315426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.316625 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.316633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:41.316833 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:41.317546 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:41.317568 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:41.318549 (+ 981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:41.322943 (+ 4394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:41.322948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:41.322954 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:41.323698 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:41.324100 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:41.324106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:41.324109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:41.324255 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:41.324914 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:41.324919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:41.325099 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 13:55:41.325231 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 13:55:41.325542 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":368,"threads_started":1}
I20260927 13:55:41.345516 23828 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:44417
I20260927 13:55:41.345558 23963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:44417 every 8 connection(s)
I20260927 13:55:41.346729 23828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:55:41.348811 23964 heartbeater.cc:347] Connected to a master server at 127.22.26.254:45575
I20260927 13:55:41.349041 23964 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:41.349686 23964 heartbeater.cc:511] Master 127.22.26.254:45575 requested a full tablet report, sending...
I20260927 13:55:41.351670 23757 ts_manager.cc:194] Registered new tserver with Master: bbb135d70fdd4a1db14249aaf892d8fc (127.22.26.193:44417)
I20260927 13:55:41.352226 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 23828
I20260927 13:55:41.352448 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:55:41.353251 22635 external_mini_cluster.cc:1428] Setting key 033fec4d017c65cf3cade58535eae855
I20260927 13:55:41.353883 23757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:35965
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:55:41.393512 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:45575
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:41.709823 23968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:41.710129 23968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:41.710307 23968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:41.717456 23968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:41.717598 23968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:41.717760 23968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:55:41.717901 23968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:55:41.727277 23968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.26.254:45575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:41.728446 23968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:41.729740 23968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:41.742343 23976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:41.742403 23973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:41.742429 23974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:41.743209 23968 server_base.cc:1034] running on GCE node
I20260927 13:55:41.743803 23968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:41.744586 23968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:41.745834 23968 hybrid_clock.cc:648] HybridClock initialized: now 1790517341745779 us; error 49 us; skew 500 ppm
I20260927 13:55:41.746206 23968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:41 dist-test-slave-gkks krb5kdc[23712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517341, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:41.755038 23968 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:55:41.756592 23968 webserver.cc:460] Webserver started at http://127.22.26.194:42091/ using document root <none> and password file <none>
I20260927 13:55:41.757279 23968 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:41.757386 23968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:41.757743 23968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:41.760586 23968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "402a1a36a7d546eca6c3656616d74784"
format_stamp: "Formatted at 2026-09-27 13:55:41 on dist-test-slave-gkks"
server_key: "6fd220da5fb55bd0da40b85cd96f9a72"
server_key_iv: "edb29524d97be8fc08f0dbbb06ae6173"
server_key_version: "encryptionkey@0"
I20260927 13:55:41.761502 23968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "402a1a36a7d546eca6c3656616d74784"
format_stamp: "Formatted at 2026-09-27 13:55:41 on dist-test-slave-gkks"
server_key: "6fd220da5fb55bd0da40b85cd96f9a72"
server_key_iv: "edb29524d97be8fc08f0dbbb06ae6173"
server_key_version: "encryptionkey@0"
I20260927 13:55:41.766537 23968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:55:41.769925 23983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:41.771224 23968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:55:41.771406 23968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "402a1a36a7d546eca6c3656616d74784"
format_stamp: "Formatted at 2026-09-27 13:55:41 on dist-test-slave-gkks"
server_key: "6fd220da5fb55bd0da40b85cd96f9a72"
server_key_iv: "edb29524d97be8fc08f0dbbb06ae6173"
server_key_version: "encryptionkey@0"
I20260927 13:55:41.771629 23968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:41.793228 23968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:41.797518 23968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:41.797791 23968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:41.800870 23968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:41.800971 23968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:41.801080 23968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:41.801122 23968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:55:41 dist-test-slave-gkks krb5kdc[23712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517341, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:41.825883 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:41.801155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:34209 (local address 127.22.26.254:45575)
0927 13:55:41.801398 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:41.801407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:41.802827 (+ 1420us) server_negotiation.cc:407] Connection header received
0927 13:55:41.805027 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:41.805033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:41.805122 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:41.805244 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:41.807538 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.808945 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.811107 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.813003 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.813155 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:41.818406 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:41.818412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:41.818415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:41.818447 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:41.821985 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:41.823030 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:41.823038 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:41.823045 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:41.823278 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:41.823786 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:41.823791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:41.823795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:41.824204 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:41.824386 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:41.825297 (+ 911us) server_negotiation.cc:299] Negotiation successful
0927 13:55:41.825535 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:55:41.827414 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:41.801777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:45575 (local address 127.22.26.194:34209)
0927 13:55:41.802620 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:41.802673 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:41.804718 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:41.805511 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:41.805526 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:41.806353 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:41.807278 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:41.807310 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.809140 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.809147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:41.810930 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:41.810943 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.813268 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.813279 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:41.813582 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:41.814412 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:41.814453 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:41.817954 (+ 3501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:41.822148 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:41.822156 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:41.822179 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:41.822854 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:41.823415 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:41.823421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:41.823424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:41.823654 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:41.824360 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:41.824369 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:41.824800 (+ 431us) client_negotiation.cc:769] Sending connection context
0927 13:55:41.825199 (+ 399us) client_negotiation.cc:240] Negotiation successful
0927 13:55:41.825782 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":386,"threads_started":1}
I20260927 13:55:41.844092 23968 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:40193
I20260927 13:55:41.844113 24102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:40193 every 8 connection(s)
I20260927 13:55:41.845453 23968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:55:41.847875 24103 heartbeater.cc:347] Connected to a master server at 127.22.26.254:45575
I20260927 13:55:41.848114 24103 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:41.848856 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:41.834320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:57879 (local address 127.22.26.254:45575)
0927 13:55:41.834550 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:41.834558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:41.835170 (+ 612us) server_negotiation.cc:407] Connection header received
0927 13:55:41.835443 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:41.835449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:41.835523 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:41.835622 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:41.836499 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.837528 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.839549 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.840105 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.840243 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:41.842775 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:41.842781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:41.842785 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:41.842821 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:41.845691 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:41.846592 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:41.846597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:41.846600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:41.846763 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:41.847250 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:41.847255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:41.847258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:41.847693 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:41.847836 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:41.848381 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 13:55:41.848556 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:55:41.848994 24103 heartbeater.cc:511] Master 127.22.26.254:45575 requested a full tablet report, sending...
I20260927 13:55:41.849035 24080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:41.834332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:45575 (local address 127.22.26.194:57879)
0927 13:55:41.835036 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:41.835060 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:41.835249 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:41.835655 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:41.835659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:41.836079 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:41.836359 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:41.836368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.837669 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.837674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:41.839389 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:41.839401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:41.840343 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:41.840350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:41.840590 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:41.841384 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:41.841418 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:41.842562 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:41.845833 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:41.845838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:41.845842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:41.846397 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:41.846934 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:41.846937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:41.846940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:41.847112 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:41.847815 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:41.847823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:41.848031 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 13:55:41.848189 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 13:55:41.848567 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":381,"threads_started":1}
I20260927 13:55:41.849105 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 23968
I20260927 13:55:41.849288 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:55:41.849804 22635 external_mini_cluster.cc:1428] Setting key 45f80af0759f71faf06a9276f345b058
I20260927 13:55:41.850528 23757 ts_manager.cc:194] Registered new tserver with Master: 402a1a36a7d546eca6c3656616d74784 (127.22.26.194:40193)
I20260927 13:55:41.851420 23757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:34209
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:55:41.890921 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:45575
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:42.215145 24107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:42.215412 24107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:42.215528 24107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:42.222687 24107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:42.222797 24107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:42.222899 24107 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:55:42.222987 24107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:55:42.233002 24107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.26.254:45575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:42.234201 24107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:42.235590 24107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:42.249038 24115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:42.249101 24113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:42.249521 24112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:42.249818 24107 server_base.cc:1034] running on GCE node
I20260927 13:55:42.250440 24107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:42.251426 24107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:42.252691 24107 hybrid_clock.cc:648] HybridClock initialized: now 1790517342252626 us; error 48 us; skew 500 ppm
I20260927 13:55:42.253012 24107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:42 dist-test-slave-gkks krb5kdc[23712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517342, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:42.256345 24107 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:55:42.257797 24107 webserver.cc:460] Webserver started at http://127.22.26.195:37101/ using document root <none> and password file <none>
I20260927 13:55:42.258496 24107 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:42.258630 24107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:42.258955 24107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:42.261945 24107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "fc05da817a8f4da9ad81d43a3c5d3d53"
format_stamp: "Formatted at 2026-09-27 13:55:42 on dist-test-slave-gkks"
server_key: "c9ed368f626be3e762ba3da860bcf52c"
server_key_iv: "dedfa1d76aa6b9a64723adc646012ea0"
server_key_version: "encryptionkey@0"
I20260927 13:55:42.262771 24107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "fc05da817a8f4da9ad81d43a3c5d3d53"
format_stamp: "Formatted at 2026-09-27 13:55:42 on dist-test-slave-gkks"
server_key: "c9ed368f626be3e762ba3da860bcf52c"
server_key_iv: "dedfa1d76aa6b9a64723adc646012ea0"
server_key_version: "encryptionkey@0"
I20260927 13:55:42.267850 24107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 13:55:42.271231 24122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:42.272455 24107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:55:42.272631 24107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "fc05da817a8f4da9ad81d43a3c5d3d53"
format_stamp: "Formatted at 2026-09-27 13:55:42 on dist-test-slave-gkks"
server_key: "c9ed368f626be3e762ba3da860bcf52c"
server_key_iv: "dedfa1d76aa6b9a64723adc646012ea0"
server_key_version: "encryptionkey@0"
I20260927 13:55:42.272859 24107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:42.287858 24107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:42.291646 24107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:42.291914 24107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:42.294598 24107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:42.294708 24107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:42.294847 24107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:42.294912 24107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:55:42 dist-test-slave-gkks krb5kdc[23712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517342, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:42.318826 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.294998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:47841 (local address 127.22.26.254:45575)
0927 13:55:42.295209 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:42.295218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:42.296577 (+ 1359us) server_negotiation.cc:407] Connection header received
0927 13:55:42.298718 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:42.298724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:42.298814 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:42.298919 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:42.301294 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.302664 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.305312 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.305921 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.306058 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.311564 (+ 5506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:42.311570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:42.311577 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:42.311625 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:42.315179 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:42.316183 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:42.316191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:42.316196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:42.316392 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:42.316995 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:42.317000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:42.317003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:42.317365 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:42.317525 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:42.318228 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 13:55:42.318463 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:55:42.319926 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.295646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:45575 (local address 127.22.26.195:47841)
0927 13:55:42.296422 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:42.296462 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:42.298392 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:42.299245 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:42.299259 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:42.300226 (+ 967us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:42.301080 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.301119 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.302842 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.302848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.305147 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.305159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.306151 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.306159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.306373 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.307243 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:42.307278 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:42.311229 (+ 3951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:42.315322 (+ 4093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:42.315331 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:42.315350 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:42.315984 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:42.316515 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:42.316520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:42.316524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:42.316876 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:42.317491 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:42.317499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:42.317854 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 13:55:42.318112 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 13:55:42.318632 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":354,"threads_started":1}
I20260927 13:55:42.340096 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.325267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:59647 (local address 127.22.26.254:45575)
0927 13:55:42.325464 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:42.325471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:42.326164 (+ 693us) server_negotiation.cc:407] Connection header received
0927 13:55:42.326335 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:42.326341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:42.326408 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:42.326501 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:42.327636 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.328927 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.330416 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.331391 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.331552 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.333232 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:42.333240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:42.333248 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:42.333296 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:42.336957 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:42.337926 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:42.337931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:42.337935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:42.338095 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:42.338557 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:42.338563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:42.338567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:42.338943 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:42.339116 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:42.339505 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 13:55:42.339718 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 13:55:42.340233 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.325299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:45575 (local address 127.22.26.195:59647)
0927 13:55:42.326072 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:42.326087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:42.326225 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:42.326521 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:42.326525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:42.326969 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:42.327387 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.327405 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.329100 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.329107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.330273 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.330285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.331573 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.331578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.331696 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.332352 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:42.332368 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:42.333029 (+ 661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:42.337125 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:42.337133 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:42.337139 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:42.337804 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:42.338250 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:42.338258 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:42.338263 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:42.338457 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:42.339050 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:42.339055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:42.339255 (+ 200us) client_negotiation.cc:769] Sending connection context
0927 13:55:42.339417 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 13:55:42.339746 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":468,"threads_started":1}
I20260927 13:55:42.354584 24107 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:33931
I20260927 13:55:42.354604 24241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:33931 every 8 connection(s)
I20260927 13:55:42.355805 24107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:55:42.357120 23964 heartbeater.cc:503] Master 127.22.26.254:45575 was elected leader, sending a full tablet report...
I20260927 13:55:42.358042 24242 heartbeater.cc:347] Connected to a master server at 127.22.26.254:45575
I20260927 13:55:42.358160 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 24107
I20260927 13:55:42.358284 24242 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:42.358361 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:55:42.358927 22635 external_mini_cluster.cc:1428] Setting key e3c71ca54841c9cd489017824a96df06
I20260927 13:55:42.359043 24242 heartbeater.cc:511] Master 127.22.26.254:45575 requested a full tablet report, sending...
I20260927 13:55:42.360559 23756 ts_manager.cc:194] Registered new tserver with Master: fc05da817a8f4da9ad81d43a3c5d3d53 (127.22.26.195:33931)
I20260927 13:55:42.361532 23756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:47841
I20260927 13:55:42.361828 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 13:55:42 dist-test-slave-gkks krb5kdc[23712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517342, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:55:42 dist-test-slave-gkks krb5kdc[23712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517342, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:42.406062 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.392808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.22.26.254:45575)
0927 13:55:42.392974 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:42.392978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:42.393282 (+ 304us) server_negotiation.cc:407] Connection header received
0927 13:55:42.393436 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:42.393440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:42.393503 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:42.393594 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:42.394976 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.395834 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.397050 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.397484 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.397580 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.401334 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:42.401339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:42.401342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:42.401373 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:42.403933 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:42.404465 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:42.404468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:42.404470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:42.404580 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:42.404900 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:42.404903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:42.404905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:42.405176 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:42.405362 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:42.405547 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 13:55:42.405788 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:55:42.410058 23756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:55:42.416816 23756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:55:42.439976 24256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.428411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:33931 (local address 127.0.0.1:40384)
0927 13:55:42.430449 (+ 2038us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:42.430480 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:42.430613 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:42.432831 (+ 2218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:42.432837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:42.432865 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:42.433281 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.433297 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.435583 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.435588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.437435 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.437449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.438891 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.438898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.439146 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.439193 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 13:55:42.439341 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 13:55:42.439521 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"mutex_wait_us":86,"thread_start_us":2865,"threads_started":1}
I20260927 13:55:42.440142 24258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.428898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.22.26.195:33931)
0927 13:55:42.431231 (+ 2333us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:42.431240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:42.431276 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:55:42.431385 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:42.431396 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:42.432627 (+ 1231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:42.433075 (+ 448us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:42.433705 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.435428 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.437620 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.438697 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.438846 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.439081 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:42.439655 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 13:55:42.439813 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2095,"thread_start_us":1943,"threads_started":1}
I20260927 13:55:42.440877 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.428613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:44417 (local address 127.0.0.1:54512)
0927 13:55:42.431530 (+ 2917us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:42.431551 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:42.431762 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:42.432753 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:42.432762 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:42.432797 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:42.433985 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.434007 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.435618 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.435623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.437877 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.437891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.440043 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.440050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.440216 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.440257 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 13:55:42.440344 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 13:55:42.440466 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2794,"mutex_wait_us":32,"thread_start_us":1441,"threads_started":1}
I20260927 13:55:42.442636 24259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.431682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:40193 (local address 127.0.0.1:44768)
0927 13:55:42.433822 (+ 2140us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:42.433841 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:42.433960 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:42.434947 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:42.434952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:42.434979 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:42.435416 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.435427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.437367 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.437374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.440415 (+ 3041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.440429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.441875 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.441881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.442058 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.442093 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:55:42.442191 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 13:55:42.442328 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1996,"thread_start_us":1848,"threads_started":1}
I20260927 13:55:42.442685 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.431829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.22.26.194:40193)
0927 13:55:42.434418 (+ 2589us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:42.434426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:42.434451 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:55:42.434531 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:42.434538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:42.434756 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:42.434934 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:42.435779 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.437209 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.440589 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.441666 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.441833 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.442018 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:42.442232 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 13:55:42.442416 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2438,"thread_start_us":2320,"threads_started":1}
I20260927 13:55:42.443113 24257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.428869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.22.26.193:44417)
0927 13:55:42.429907 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:42.429914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:42.431647 (+ 1733us) server_negotiation.cc:407] Connection header received
0927 13:55:42.431951 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:42.431960 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:42.432206 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:42.432391 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:42.434211 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.435433 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.438039 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.439556 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.439694 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.439893 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:42.440576 (+ 683us) server_negotiation.cc:299] Negotiation successful
0927 13:55:42.440766 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":764,"threads_started":1}
I20260927 13:55:42.443775 24172 tablet_service.cc:1467] Processing CreateTablet for tablet a5fb5b24adb54be1ac7a38b39cb1a515 (DEFAULT_TABLE table=test-table [id=5abf9af20b574139a2965563e520ea74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:42.445313 24172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5fb5b24adb54be1ac7a38b39cb1a515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:42.446256 23898 tablet_service.cc:1467] Processing CreateTablet for tablet a5fb5b24adb54be1ac7a38b39cb1a515 (DEFAULT_TABLE table=test-table [id=5abf9af20b574139a2965563e520ea74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:42.446240 24033 tablet_service.cc:1467] Processing CreateTablet for tablet a5fb5b24adb54be1ac7a38b39cb1a515 (DEFAULT_TABLE table=test-table [id=5abf9af20b574139a2965563e520ea74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:42.447531 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5fb5b24adb54be1ac7a38b39cb1a515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:42.447544 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5fb5b24adb54be1ac7a38b39cb1a515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:42.456081 24261 tablet_bootstrap.cc:492] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53: Bootstrap starting.
I20260927 13:55:42.459842 24261 tablet_bootstrap.cc:654] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:42.460803 24261 log.cc:824] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:42.461684 24262 tablet_bootstrap.cc:492] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc: Bootstrap starting.
I20260927 13:55:42.462488 24263 tablet_bootstrap.cc:492] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784: Bootstrap starting.
I20260927 13:55:42.465430 24262 tablet_bootstrap.cc:654] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:42.465627 24261 tablet_bootstrap.cc:492] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53: No bootstrap required, opened a new log
I20260927 13:55:42.465719 24263 tablet_bootstrap.cc:654] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:42.465881 24261 ts_tablet_manager.cc:1397] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 13:55:42.466662 24262 log.cc:824] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:42.467391 24263 log.cc:824] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:42.469231 24262 tablet_bootstrap.cc:492] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc: No bootstrap required, opened a new log
I20260927 13:55:42.469483 24262 ts_tablet_manager.cc:1397] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 13:55:42.470114 24263 tablet_bootstrap.cc:492] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784: No bootstrap required, opened a new log
I20260927 13:55:42.470347 24263 ts_tablet_manager.cc:1397] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260927 13:55:42.476126 24261 raft_consensus.cc:348] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } }
I20260927 13:55:42.476555 24261 raft_consensus.cc:374] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:42.476666 24261 raft_consensus.cc:729] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc05da817a8f4da9ad81d43a3c5d3d53, State: Initialized, Role: FOLLOWER
I20260927 13:55:42.476816 24262 raft_consensus.cc:348] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } }
I20260927 13:55:42.477209 24262 raft_consensus.cc:374] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:42.477303 24262 raft_consensus.cc:729] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb135d70fdd4a1db14249aaf892d8fc, State: Initialized, Role: FOLLOWER
I20260927 13:55:42.477883 24262 consensus_queue.cc:260] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } }
I20260927 13:55:42.479633 24262 ts_tablet_manager.cc:1428] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260927 13:55:42.480343 24261 consensus_queue.cc:260] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } }
I20260927 13:55:42.480847 24267 raft_consensus.cc:482] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:55:42.481120 24267 raft_consensus.cc:504] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } }
I20260927 13:55:42.481417 24242 heartbeater.cc:503] Master 127.22.26.254:45575 was elected leader, sending a full tablet report...
I20260927 13:55:42.483821 24263 raft_consensus.cc:348] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } }
I20260927 13:55:42.484216 24263 raft_consensus.cc:374] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:42.484319 24263 raft_consensus.cc:729] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 402a1a36a7d546eca6c3656616d74784, State: Initialized, Role: FOLLOWER
I20260927 13:55:42.484989 24263 consensus_queue.cc:260] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } }
I20260927 13:55:42.486017 24103 heartbeater.cc:503] Master 127.22.26.254:45575 was elected leader, sending a full tablet report...
I20260927 13:55:42.488086 24261 ts_tablet_manager.cc:1428] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53: Time spent starting tablet: real 0.022s user 0.010s sys 0.006s
I20260927 13:55:42.489563 24263 ts_tablet_manager.cc:1428] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784: Time spent starting tablet: real 0.019s user 0.014s sys 0.001s
I20260927 13:55:42.490392 24267 leader_election.cc:288] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbb135d70fdd4a1db14249aaf892d8fc (127.22.26.193:44417), 402a1a36a7d546eca6c3656616d74784 (127.22.26.194:40193)
I20260927 13:55:42.492286 24257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.484736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:40139 (local address 127.22.26.193:44417)
0927 13:55:42.484920 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:42.484926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:42.484945 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:55:42.485012 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:42.485016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:42.485084 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:42.485185 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:42.485631 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.486622 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.490311 (+ 3689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.491554 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.491690 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.491740 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:42.492043 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 13:55:42.492126 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:55:42.492440 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.484491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:44417 (local address 127.22.26.195:40139)
0927 13:55:42.484683 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:42.484704 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:42.484812 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:42.485195 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:42.485198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:42.485219 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:42.485445 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.485460 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.486792 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.486798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.488958 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.488971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.491690 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.491696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.491886 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.491921 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:55:42.492027 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 13:55:42.492157 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 13:55:42.493256 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5fb5b24adb54be1ac7a38b39cb1a515" candidate_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" is_pre_election: true
I20260927 13:55:42.493619 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.486445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:52631 (local address 127.22.26.194:40193)
0927 13:55:42.486614 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:42.486621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:42.486643 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:55:42.486852 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:42.486858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:42.486929 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:42.486988 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:42.487884 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.488949 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.491597 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.492821 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.492934 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.492978 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:42.493337 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 13:55:42.493420 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:55:42.493760 23919 raft_consensus.cc:2454] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc05da817a8f4da9ad81d43a3c5d3d53 in term 0.
I20260927 13:55:42.493836 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.486017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:40193 (local address 127.22.26.195:52631)
0927 13:55:42.486538 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:42.486556 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:42.486668 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:42.487180 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:42.487184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:42.487206 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:42.487678 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.487690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.489174 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.489178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.491446 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.491463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.492997 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.493004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.493179 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.493211 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 13:55:42.493323 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 13:55:42.493458 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":636,"threads_started":1}
I20260927 13:55:42.494379 24124 leader_election.cc:302] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbb135d70fdd4a1db14249aaf892d8fc, fc05da817a8f4da9ad81d43a3c5d3d53; no voters:
I20260927 13:55:42.494724 24267 raft_consensus.cc:2781] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:55:42.494709 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5fb5b24adb54be1ac7a38b39cb1a515" candidate_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "402a1a36a7d546eca6c3656616d74784" is_pre_election: true
I20260927 13:55:42.494899 24267 raft_consensus.cc:482] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:55:42.495002 24267 raft_consensus.cc:3037] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:42.495221 24053 raft_consensus.cc:2454] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc05da817a8f4da9ad81d43a3c5d3d53 in term 0.
I20260927 13:55:42.497411 24267 raft_consensus.cc:504] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } }
I20260927 13:55:42.497939 24267 leader_election.cc:288] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [CANDIDATE]: Term 1 election: Requested vote from peers bbb135d70fdd4a1db14249aaf892d8fc (127.22.26.193:44417), 402a1a36a7d546eca6c3656616d74784 (127.22.26.194:40193)
I20260927 13:55:42.498373 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5fb5b24adb54be1ac7a38b39cb1a515" candidate_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "402a1a36a7d546eca6c3656616d74784"
I20260927 13:55:42.498353 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5fb5b24adb54be1ac7a38b39cb1a515" candidate_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb135d70fdd4a1db14249aaf892d8fc"
I20260927 13:55:42.498553 24053 raft_consensus.cc:3037] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:42.498553 23919 raft_consensus.cc:3037] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:42.501652 24053 raft_consensus.cc:2454] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc05da817a8f4da9ad81d43a3c5d3d53 in term 1.
I20260927 13:55:42.501652 23919 raft_consensus.cc:2454] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc05da817a8f4da9ad81d43a3c5d3d53 in term 1.
I20260927 13:55:42.502015 24126 leader_election.cc:302] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 402a1a36a7d546eca6c3656616d74784, fc05da817a8f4da9ad81d43a3c5d3d53; no voters:
I20260927 13:55:42.502288 24267 raft_consensus.cc:2781] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:42.502441 24267 raft_consensus.cc:686] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [term 1 LEADER]: Becoming Leader. State: Replica: fc05da817a8f4da9ad81d43a3c5d3d53, State: Running, Role: LEADER
I20260927 13:55:42.502861 24267 consensus_queue.cc:237] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } }
I20260927 13:55:42.506876 23757 catalog_manager.cc:5495] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc05da817a8f4da9ad81d43a3c5d3d53 (127.22.26.195). New cstate: current_term: 1 leader_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } health_report { overall_health: UNKNOWN } } }
I20260927 13:55:42.548085 24258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.542378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.22.26.195:33931)
0927 13:55:42.542551 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:42.542556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:42.542699 (+ 143us) server_negotiation.cc:407] Connection header received
0927 13:55:42.543009 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:42.543013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:42.543135 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:42.543249 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:42.543613 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.544328 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.545815 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.546286 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.546381 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.546904 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:42.547451 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:42.547557 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:42.547816 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 13:55:42.547884 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:55:42.556944 23919 raft_consensus.cc:1261] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc [term 1 FOLLOWER]: Refusing update from remote peer fc05da817a8f4da9ad81d43a3c5d3d53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:42.556944 24053 raft_consensus.cc:1261] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784 [term 1 FOLLOWER]: Refusing update from remote peer fc05da817a8f4da9ad81d43a3c5d3d53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:42.558103 24267 consensus_queue.cc:1035] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:42.558458 24269 consensus_queue.cc:1035] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:42.571780 24274 mvcc.cc:205] Tried to move back new op lower bound from 7333959035103096832 to 7333959034895892480. Current Snapshot: MvccSnapshot[applied={T|T < 7333959035103096832}]
I20260927 13:55:42.595943 23757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:55:42.596225 23757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:55:42.599685 23757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003: Sending DeleteTablet for 3 replicas of tablet a5fb5b24adb54be1ac7a38b39cb1a515
I20260927 13:55:42.600814 24172 tablet_service.cc:1514] Processing DeleteTablet for tablet a5fb5b24adb54be1ac7a38b39cb1a515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:42 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:40384
I20260927 13:55:42.600814 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet a5fb5b24adb54be1ac7a38b39cb1a515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:42 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:54512
I20260927 13:55:42.600924 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet a5fb5b24adb54be1ac7a38b39cb1a515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:42 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:44768
I20260927 13:55:42.602890 23757 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:55:42.603965 23757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:55:42.613212 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 14435933c2274113b07899c35c0c1d3c (DEFAULT_TABLE table=test-table [id=d18f0cd6cd914d03a82b13acff4da318]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:42.613482 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 14435933c2274113b07899c35c0c1d3c (DEFAULT_TABLE table=test-table [id=d18f0cd6cd914d03a82b13acff4da318]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:42.613994 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14435933c2274113b07899c35c0c1d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:42.614037 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14435933c2274113b07899c35c0c1d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:42.614357 24284 tablet_replica.cc:330] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc: stopping tablet replica
I20260927 13:55:42.614769 24284 raft_consensus.cc:2229] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:55:42.615306 24284 raft_consensus.cc:2258] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:42.618137 24284 ts_tablet_manager.cc:1905] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:42.620045 24171 tablet_service.cc:1467] Processing CreateTablet for tablet 14435933c2274113b07899c35c0c1d3c (DEFAULT_TABLE table=test-table [id=d18f0cd6cd914d03a82b13acff4da318]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:42.620735 24171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14435933c2274113b07899c35c0c1d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:42.622448 24285 tablet_replica.cc:330] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53: stopping tablet replica
I20260927 13:55:42.622685 24262 tablet_bootstrap.cc:492] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc: Bootstrap starting.
I20260927 13:55:42.622694 24286 tablet_replica.cc:330] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784: stopping tablet replica
I20260927 13:55:42.622875 24285 raft_consensus.cc:2229] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:55:42.623113 24286 raft_consensus.cc:2229] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:55:42.623566 24286 raft_consensus.cc:2258] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:42.624225 24262 tablet_bootstrap.cc:654] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:42.625397 24286 ts_tablet_manager.cc:1905] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:42.627919 24261 tablet_bootstrap.cc:492] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53: Bootstrap starting.
I20260927 13:55:42.629348 24262 tablet_bootstrap.cc:492] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc: No bootstrap required, opened a new log
I20260927 13:55:42.629494 24261 tablet_bootstrap.cc:654] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:42.629495 24262 ts_tablet_manager.cc:1397] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 13:55:42.630285 24262 raft_consensus.cc:348] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:42.630517 24262 raft_consensus.cc:374] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:42.630615 24262 raft_consensus.cc:729] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb135d70fdd4a1db14249aaf892d8fc, State: Initialized, Role: FOLLOWER
I20260927 13:55:42.630750 24263 tablet_bootstrap.cc:492] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784: Bootstrap starting.
I20260927 13:55:42.630939 24262 consensus_queue.cc:260] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:42.631511 24262 ts_tablet_manager.cc:1428] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 13:55:42.633364 24286 ts_tablet_manager.cc:1918] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:55:42.633523 24286 log.cc:1196] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/a5fb5b24adb54be1ac7a38b39cb1a515
I20260927 13:55:42.633731 24263 tablet_bootstrap.cc:654] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:42.634145 24286 ts_tablet_manager.cc:1939] T a5fb5b24adb54be1ac7a38b39cb1a515 P 402a1a36a7d546eca6c3656616d74784: Deleting consensus metadata
I20260927 13:55:42.627692 24285 raft_consensus.cc:2258] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:42.635996 23744 catalog_manager.cc:4841] TS 402a1a36a7d546eca6c3656616d74784 (127.22.26.194:40193): tablet a5fb5b24adb54be1ac7a38b39cb1a515 (table test-table [id=5abf9af20b574139a2965563e520ea74]) successfully deleted
I20260927 13:55:42.637858 24284 ts_tablet_manager.cc:1918] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:55:42.638024 24284 log.cc:1196] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/a5fb5b24adb54be1ac7a38b39cb1a515
I20260927 13:55:42.638549 24261 tablet_bootstrap.cc:492] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53: No bootstrap required, opened a new log
I20260927 13:55:42.638566 24284 ts_tablet_manager.cc:1939] T a5fb5b24adb54be1ac7a38b39cb1a515 P bbb135d70fdd4a1db14249aaf892d8fc: Deleting consensus metadata
I20260927 13:55:42.638705 24261 ts_tablet_manager.cc:1397] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260927 13:55:42.639698 24261 raft_consensus.cc:348] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:42.640282 23742 catalog_manager.cc:4841] TS bbb135d70fdd4a1db14249aaf892d8fc (127.22.26.193:44417): tablet a5fb5b24adb54be1ac7a38b39cb1a515 (table test-table [id=5abf9af20b574139a2965563e520ea74]) successfully deleted
I20260927 13:55:42.640501 24261 raft_consensus.cc:374] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:42.640642 24261 raft_consensus.cc:729] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc05da817a8f4da9ad81d43a3c5d3d53, State: Initialized, Role: FOLLOWER
I20260927 13:55:42.640946 24261 consensus_queue.cc:260] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:42.641438 24263 tablet_bootstrap.cc:492] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784: No bootstrap required, opened a new log
I20260927 13:55:42.641636 24263 ts_tablet_manager.cc:1397] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260927 13:55:42.642020 24285 ts_tablet_manager.cc:1905] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:42.642457 24261 ts_tablet_manager.cc:1428] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 13:55:42.643162 24263 raft_consensus.cc:348] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:42.643404 24263 raft_consensus.cc:374] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:42.643518 24263 raft_consensus.cc:729] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 402a1a36a7d546eca6c3656616d74784, State: Initialized, Role: FOLLOWER
I20260927 13:55:42.643848 24263 consensus_queue.cc:260] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:42.645282 24263 ts_tablet_manager.cc:1428] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 13:55:42.649818 24285 ts_tablet_manager.cc:1918] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:55:42.649989 24285 log.cc:1196] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/a5fb5b24adb54be1ac7a38b39cb1a515
I20260927 13:55:42.650574 24285 ts_tablet_manager.cc:1939] T a5fb5b24adb54be1ac7a38b39cb1a515 P fc05da817a8f4da9ad81d43a3c5d3d53: Deleting consensus metadata
I20260927 13:55:42.652243 23743 catalog_manager.cc:4841] TS fc05da817a8f4da9ad81d43a3c5d3d53 (127.22.26.195:33931): tablet a5fb5b24adb54be1ac7a38b39cb1a515 (table test-table [id=5abf9af20b574139a2965563e520ea74]) successfully deleted
W20260927 13:55:42.848570 24104 tablet.cc:2369] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:55:42.849925 23965 tablet.cc:2369] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:55:42.858405 24243 tablet.cc:2369] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:55:42.984246 24268 raft_consensus.cc:482] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:55:42.984470 24268 raft_consensus.cc:504] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:42.986057 24268 leader_election.cc:288] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 402a1a36a7d546eca6c3656616d74784 (127.22.26.194:40193), fc05da817a8f4da9ad81d43a3c5d3d53 (127.22.26.195:33931)
I20260927 13:55:42.995028 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.986547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:37605 (local address 127.22.26.194:40193)
0927 13:55:42.986735 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:42.986741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:42.988251 (+ 1510us) server_negotiation.cc:407] Connection header received
0927 13:55:42.988437 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:42.988442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:42.988522 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:42.988623 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:42.989344 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.990286 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.993383 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.994112 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.994275 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.994348 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:42.994618 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 13:55:42.994751 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:55:42.995115 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.986650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:40193 (local address 127.22.26.193:37605)
0927 13:55:42.988129 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:42.988152 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:42.988289 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:42.988656 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:42.988660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:42.988682 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:42.989178 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.989187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.990407 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.990413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.993252 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.993261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.994283 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.994289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.994468 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.994492 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 13:55:42.994592 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 13:55:42.994719 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"mutex_wait_us":82,"thread_start_us":1133,"threads_started":1}
I20260927 13:55:42.995771 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14435933c2274113b07899c35c0c1d3c" candidate_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "402a1a36a7d546eca6c3656616d74784" is_pre_election: true
I20260927 13:55:42.996033 24053 raft_consensus.cc:2454] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbb135d70fdd4a1db14249aaf892d8fc in term 0.
I20260927 13:55:42.996177 24258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.986961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:46923 (local address 127.22.26.195:33931)
0927 13:55:42.987160 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:42.987166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:42.987794 (+ 628us) server_negotiation.cc:407] Connection header received
0927 13:55:42.988133 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:42.988146 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:42.988302 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:42.988406 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:42.989150 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.990427 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.993199 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.994714 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.994853 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.994923 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:42.995780 (+ 857us) server_negotiation.cc:299] Negotiation successful
0927 13:55:42.995916 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:55:42.996516 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:42.986477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:33931 (local address 127.22.26.193:46923)
0927 13:55:42.987691 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:42.987710 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:42.987918 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:42.988454 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:42.988459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:42.988485 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:42.988962 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.988974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.990580 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.990586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.993017 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:42.993033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:42.995075 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:42.995424 (+ 349us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:42.995602 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:42.995636 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 13:55:42.995736 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 13:55:42.995854 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"mutex_wait_us":64,"thread_start_us":896,"threads_started":1}
I20260927 13:55:42.996502 23847 leader_election.cc:302] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 402a1a36a7d546eca6c3656616d74784, bbb135d70fdd4a1db14249aaf892d8fc; no voters:
I20260927 13:55:42.996927 24268 raft_consensus.cc:2781] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:55:42.997040 24268 raft_consensus.cc:482] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:55:42.997124 24268 raft_consensus.cc:3037] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:43.000386 24268 raft_consensus.cc:504] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:43.000576 24197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14435933c2274113b07899c35c0c1d3c" candidate_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" is_pre_election: true
I20260927 13:55:43.000978 24197 raft_consensus.cc:2454] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbb135d70fdd4a1db14249aaf892d8fc in term 0.
I20260927 13:55:43.001108 24268 leader_election.cc:288] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [CANDIDATE]: Term 1 election: Requested vote from peers 402a1a36a7d546eca6c3656616d74784 (127.22.26.194:40193), fc05da817a8f4da9ad81d43a3c5d3d53 (127.22.26.195:33931)
I20260927 13:55:43.001871 24197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14435933c2274113b07899c35c0c1d3c" candidate_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53"
I20260927 13:55:43.002096 24197 raft_consensus.cc:3037] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:43.002146 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14435933c2274113b07899c35c0c1d3c" candidate_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "402a1a36a7d546eca6c3656616d74784"
I20260927 13:55:43.002388 24053 raft_consensus.cc:3037] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:43.004426 24053 raft_consensus.cc:2454] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbb135d70fdd4a1db14249aaf892d8fc in term 1.
I20260927 13:55:43.004796 23847 leader_election.cc:302] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 402a1a36a7d546eca6c3656616d74784, bbb135d70fdd4a1db14249aaf892d8fc; no voters:
I20260927 13:55:43.005055 24268 raft_consensus.cc:2781] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:43.005239 24197 raft_consensus.cc:2454] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbb135d70fdd4a1db14249aaf892d8fc in term 1.
I20260927 13:55:43.005754 24268 raft_consensus.cc:686] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [term 1 LEADER]: Becoming Leader. State: Replica: bbb135d70fdd4a1db14249aaf892d8fc, State: Running, Role: LEADER
I20260927 13:55:43.006129 24268 consensus_queue.cc:237] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:43.009411 23756 catalog_manager.cc:5495] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc reported cstate change: term changed from 0 to 1, leader changed from <none> to bbb135d70fdd4a1db14249aaf892d8fc (127.22.26.193). New cstate: current_term: 1 leader_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } health_report { overall_health: UNKNOWN } } }
Sep 27 13:55:43 dist-test-slave-gkks krb5kdc[23712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517340, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:43.106776 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.093258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:45575 (local address 127.0.0.1:46812)
0927 13:55:43.093947 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:43.093964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:43.094093 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:43.094401 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:43.094404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:43.095050 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:43.095295 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:43.095304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.096277 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.096280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:43.097226 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:43.097239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.097873 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.097878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:43.097991 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.098592 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:43.098609 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:43.101010 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:43.104628 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:43.104637 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:43.104643 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:43.105026 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:43.105348 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:43.105351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:43.105353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:43.105485 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:43.105996 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:43.106001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:43.106134 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 13:55:43.106265 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 13:55:43.106476 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":412,"threads_started":1}
I20260927 13:55:43.106871 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.093242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.22.26.254:45575)
0927 13:55:43.093906 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:43.093911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:43.094042 (+ 131us) server_negotiation.cc:407] Connection header received
0927 13:55:43.094190 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:43.094194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:43.094251 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:43.094357 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:43.095432 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.096166 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.097350 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.097731 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.097820 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.101218 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:43.101223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:43.101226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:43.101254 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:43.104513 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:43.105137 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:43.105141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:43.105143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:43.105249 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:43.105595 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:43.105599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:43.105601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:43.105872 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:43.106006 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:43.106378 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 13:55:43.106604 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":515,"thread_start_us":351,"threads_started":1}
I20260927 13:55:43.111840 23756 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:46812:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 13:55:43.112800 23756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:55:43.119007 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 43cd94a340404487afcc50585c2e23ec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6aadaa09d4b84013a48f7585aeb96d9f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:55:43.119248 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 43cd94a340404487afcc50585c2e23ec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6aadaa09d4b84013a48f7585aeb96d9f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:55:43.119592 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43cd94a340404487afcc50585c2e23ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:43.119407 24171 tablet_service.cc:1467] Processing CreateTablet for tablet 43cd94a340404487afcc50585c2e23ec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6aadaa09d4b84013a48f7585aeb96d9f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:55:43.119807 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43cd94a340404487afcc50585c2e23ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:43.119940 24171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43cd94a340404487afcc50585c2e23ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:43.124948 24263 tablet_bootstrap.cc:492] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784: Bootstrap starting.
I20260927 13:55:43.126556 24262 tablet_bootstrap.cc:492] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc: Bootstrap starting.
I20260927 13:55:43.126610 24261 tablet_bootstrap.cc:492] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53: Bootstrap starting.
I20260927 13:55:43.126609 24263 tablet_bootstrap.cc:654] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:43.128103 24262 tablet_bootstrap.cc:654] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:43.128324 24261 tablet_bootstrap.cc:654] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:43.128851 24263 tablet_bootstrap.cc:492] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784: No bootstrap required, opened a new log
I20260927 13:55:43.129001 24263 ts_tablet_manager.cc:1397] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 13:55:43.129905 24263 raft_consensus.cc:348] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:43.130117 24262 tablet_bootstrap.cc:492] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc: No bootstrap required, opened a new log
I20260927 13:55:43.130304 24262 ts_tablet_manager.cc:1397] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 13:55:43.130256 24263 raft_consensus.cc:374] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:43.130398 24263 raft_consensus.cc:729] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 402a1a36a7d546eca6c3656616d74784, State: Initialized, Role: FOLLOWER
I20260927 13:55:43.130659 24263 consensus_queue.cc:260] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:43.130923 24261 tablet_bootstrap.cc:492] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53: No bootstrap required, opened a new log
I20260927 13:55:43.130923 24262 raft_consensus.cc:348] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:43.131103 24261 ts_tablet_manager.cc:1397] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 13:55:43.131191 24262 raft_consensus.cc:374] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:43.131268 24262 raft_consensus.cc:729] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb135d70fdd4a1db14249aaf892d8fc, State: Initialized, Role: FOLLOWER
I20260927 13:55:43.131124 24270 tablet_replica.cc:439] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } }
I20260927 13:55:43.131455 24270 tablet_replica.cc:442] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:43.131492 24262 consensus_queue.cc:260] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:43.131811 24261 raft_consensus.cc:348] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:43.132045 24262 ts_tablet_manager.cc:1428] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 13:55:43.132035 24261 raft_consensus.cc:374] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:43.132118 24261 raft_consensus.cc:729] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc05da817a8f4da9ad81d43a3c5d3d53, State: Initialized, Role: FOLLOWER
I20260927 13:55:43.132349 24261 consensus_queue.cc:260] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:43.132485 24268 tablet_replica.cc:439] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } }
I20260927 13:55:43.132779 24268 tablet_replica.cc:442] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:43.132754 24282 tablet_replica.cc:439] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } }
I20260927 13:55:43.133070 24282 tablet_replica.cc:442] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:43.133606 24263 ts_tablet_manager.cc:1428] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260927 13:55:43.134109 24261 ts_tablet_manager.cc:1428] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 13:55:43.162555 24270 raft_consensus.cc:482] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:55:43.162724 24270 raft_consensus.cc:504] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:43.164183 24270 leader_election.cc:288] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbb135d70fdd4a1db14249aaf892d8fc (127.22.26.193:44417), fc05da817a8f4da9ad81d43a3c5d3d53 (127.22.26.195:33931)
I20260927 13:55:43.171157 24258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.164643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:60713 (local address 127.22.26.195:33931)
0927 13:55:43.164830 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:43.164836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:43.165744 (+ 908us) server_negotiation.cc:407] Connection header received
0927 13:55:43.165908 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:43.165913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:43.165992 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:43.166091 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:43.166633 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.167498 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.169374 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.170172 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.170336 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.170423 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:43.170725 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 13:55:43.170871 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 13:55:43.171161 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.164488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:36691 (local address 127.22.26.193:44417)
0927 13:55:43.165507 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:43.165514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:43.165535 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:55:43.165609 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:43.165614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:43.165712 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:43.165817 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:43.166452 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.167479 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.169298 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.170090 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.170212 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.170308 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:43.170665 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 13:55:43.170810 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":774,"threads_started":1}
I20260927 13:55:43.171264 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.164375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:44417 (local address 127.22.26.194:36691)
0927 13:55:43.165196 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:43.165221 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:43.165396 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:43.165860 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:43.165868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:43.165904 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:43.166290 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:43.166302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.167621 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.167630 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:43.169144 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:43.169159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.170269 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.170275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:43.170493 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.170527 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 13:55:43.170601 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 13:55:43.170746 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":39,"thread_start_us":1071,"threads_started":1}
I20260927 13:55:43.171298 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.164553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:33931 (local address 127.22.26.194:60713)
0927 13:55:43.165637 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:43.165653 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:43.165778 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:43.166114 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:43.166117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:43.166137 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:43.166476 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:43.166490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.167654 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.167659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:43.169207 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:43.169241 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.170428 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.170433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:43.170549 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.170571 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 13:55:43.170681 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 13:55:43.170809 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":9,"thread_start_us":372,"threads_started":1}
I20260927 13:55:43.171937 24197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43cd94a340404487afcc50585c2e23ec" candidate_uuid: "402a1a36a7d546eca6c3656616d74784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" is_pre_election: true
I20260927 13:55:43.171953 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43cd94a340404487afcc50585c2e23ec" candidate_uuid: "402a1a36a7d546eca6c3656616d74784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" is_pre_election: true
I20260927 13:55:43.172209 24197 raft_consensus.cc:2454] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 402a1a36a7d546eca6c3656616d74784 in term 0.
I20260927 13:55:43.172205 23919 raft_consensus.cc:2454] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 402a1a36a7d546eca6c3656616d74784 in term 0.
I20260927 13:55:43.172717 23986 leader_election.cc:302] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 402a1a36a7d546eca6c3656616d74784, bbb135d70fdd4a1db14249aaf892d8fc, fc05da817a8f4da9ad81d43a3c5d3d53; no voters:
I20260927 13:55:43.173108 24270 raft_consensus.cc:2781] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:55:43.173233 24270 raft_consensus.cc:482] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:55:43.173336 24270 raft_consensus.cc:3037] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:43.175707 24270 raft_consensus.cc:504] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:43.176270 24270 leader_election.cc:288] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [CANDIDATE]: Term 1 election: Requested vote from peers bbb135d70fdd4a1db14249aaf892d8fc (127.22.26.193:44417), fc05da817a8f4da9ad81d43a3c5d3d53 (127.22.26.195:33931)
I20260927 13:55:43.176693 24197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43cd94a340404487afcc50585c2e23ec" candidate_uuid: "402a1a36a7d546eca6c3656616d74784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53"
I20260927 13:55:43.176741 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43cd94a340404487afcc50585c2e23ec" candidate_uuid: "402a1a36a7d546eca6c3656616d74784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb135d70fdd4a1db14249aaf892d8fc"
I20260927 13:55:43.176970 24197 raft_consensus.cc:3037] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:43.177006 23919 raft_consensus.cc:3037] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:43.179288 24197 raft_consensus.cc:2454] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 402a1a36a7d546eca6c3656616d74784 in term 1.
I20260927 13:55:43.179579 23919 raft_consensus.cc:2454] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 402a1a36a7d546eca6c3656616d74784 in term 1.
I20260927 13:55:43.179679 23986 leader_election.cc:302] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 402a1a36a7d546eca6c3656616d74784, fc05da817a8f4da9ad81d43a3c5d3d53; no voters:
I20260927 13:55:43.179986 24270 raft_consensus.cc:2781] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:43.180399 24270 raft_consensus.cc:686] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [term 1 LEADER]: Becoming Leader. State: Replica: 402a1a36a7d546eca6c3656616d74784, State: Running, Role: LEADER
I20260927 13:55:43.180763 24270 consensus_queue.cc:237] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } }
I20260927 13:55:43.183600 24307 tablet_replica.cc:439] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784: TxnStatusTablet state changed. Reason: New leader 402a1a36a7d546eca6c3656616d74784. Latest consensus state: current_term: 1 leader_uuid: "402a1a36a7d546eca6c3656616d74784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } }
I20260927 13:55:43.183898 24307 tablet_replica.cc:442] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:55:43.185256 24309 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 13:55:43.185478 24309 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 13:55:43.185295 23755 catalog_manager.cc:5495] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 reported cstate change: term changed from 0 to 1, leader changed from <none> to 402a1a36a7d546eca6c3656616d74784 (127.22.26.194). New cstate: current_term: 1 leader_uuid: "402a1a36a7d546eca6c3656616d74784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } health_report { overall_health: UNKNOWN } } }
I20260927 13:55:43.207998 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.201551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:40193 (local address 127.0.0.1:44776)
0927 13:55:43.201782 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:43.201807 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:43.201918 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:43.202409 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:43.202413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:43.202423 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:55:43.202560 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:43.202567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.204022 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.204026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:43.205007 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:43.205024 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.206068 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.206073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:43.206266 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.206337 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:55:43.207518 (+ 1181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:55:43.207541 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 13:55:43.207622 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 13:55:43.207752 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 13:55:43.208091 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.201545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.22.26.194:40193)
0927 13:55:43.201762 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:43.201768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:43.201875 (+ 107us) server_negotiation.cc:407] Connection header received
0927 13:55:43.202090 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:43.202095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:43.202232 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:43.202394 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:43.202689 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.203887 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.205155 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.205879 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.206052 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.206536 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:43.207370 (+ 834us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:43.207515 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:43.207675 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 13:55:43.207776 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 13:55:43.212472 23919 raft_consensus.cc:1261] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc [term 1 FOLLOWER]: Refusing update from remote peer 402a1a36a7d546eca6c3656616d74784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:43.212999 24197 raft_consensus.cc:1261] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53 [term 1 FOLLOWER]: Refusing update from remote peer 402a1a36a7d546eca6c3656616d74784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:43.213233 24307 consensus_queue.cc:1035] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:43.213650 24270 consensus_queue.cc:1035] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:43.221238 24268 tablet_replica.cc:439] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc: TxnStatusTablet state changed. Reason: New leader 402a1a36a7d546eca6c3656616d74784. Latest consensus state: current_term: 1 leader_uuid: "402a1a36a7d546eca6c3656616d74784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } }
I20260927 13:55:43.221549 24268 tablet_replica.cc:442] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:43.222551 24282 tablet_replica.cc:439] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53: TxnStatusTablet state changed. Reason: New leader 402a1a36a7d546eca6c3656616d74784. Latest consensus state: current_term: 1 leader_uuid: "402a1a36a7d546eca6c3656616d74784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } }
I20260927 13:55:43.222877 24282 tablet_replica.cc:442] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:43.225755 24312 mvcc.cc:205] Tried to move back new op lower bound from 7333959037790953472 to 7333959037671567360. Current Snapshot: MvccSnapshot[applied={T|T < 7333959037790953472}]
I20260927 13:55:43.231739 24307 tablet_replica.cc:439] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "402a1a36a7d546eca6c3656616d74784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } }
I20260927 13:55:43.232007 24307 tablet_replica.cc:442] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:55:43.233381 24268 tablet_replica.cc:439] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "402a1a36a7d546eca6c3656616d74784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } }
I20260927 13:55:43.233659 24268 tablet_replica.cc:442] T 43cd94a340404487afcc50585c2e23ec P bbb135d70fdd4a1db14249aaf892d8fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:43.234190 24270 tablet_replica.cc:439] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "402a1a36a7d546eca6c3656616d74784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } }
I20260927 13:55:43.234732 24270 tablet_replica.cc:442] T 43cd94a340404487afcc50585c2e23ec P 402a1a36a7d546eca6c3656616d74784: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:55:43.239804 24282 tablet_replica.cc:439] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "402a1a36a7d546eca6c3656616d74784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb135d70fdd4a1db14249aaf892d8fc" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 44417 } } peers { permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 } } peers { permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 } } }
I20260927 13:55:43.240048 24282 tablet_replica.cc:442] T 43cd94a340404487afcc50585c2e23ec P fc05da817a8f4da9ad81d43a3c5d3d53: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:43.250845 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.244089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.22.26.193:44417)
0927 13:55:43.244360 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:43.244369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:43.244923 (+ 554us) server_negotiation.cc:407] Connection header received
0927 13:55:43.245154 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:43.245160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:43.245290 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:43.245450 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:43.245904 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.247051 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.248407 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.249049 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.249179 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.249547 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:43.250270 (+ 723us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:43.250420 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:43.250469 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 13:55:43.250551 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:55:43.267009 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.261299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:34445 (local address 127.22.26.194:40193)
0927 13:55:43.261487 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:43.261491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:43.262015 (+ 524us) server_negotiation.cc:407] Connection header received
0927 13:55:43.262268 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:43.262272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:43.262351 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:43.262453 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:43.262972 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.263838 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.265181 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.265686 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.265846 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.266239 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:43.266356 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:43.266452 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:43.266586 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 13:55:43.266668 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 13:55:43.267136 24325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.261314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:40193 (local address 127.22.26.193:34445)
0927 13:55:43.261909 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:43.261924 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:43.262078 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:43.262501 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:43.262505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:43.262518 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:55:43.262815 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:43.262824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.263968 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.263976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:43.265040 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:43.265052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.265875 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.265882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:43.266017 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.266069 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:55:43.266455 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:55:43.266469 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 13:55:43.266563 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 13:55:43.266704 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":320,"threads_started":1}
I20260927 13:55:43.276571 24197 raft_consensus.cc:1261] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53 [term 1 FOLLOWER]: Refusing update from remote peer bbb135d70fdd4a1db14249aaf892d8fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:43.276597 24053 raft_consensus.cc:1261] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784 [term 1 FOLLOWER]: Refusing update from remote peer bbb135d70fdd4a1db14249aaf892d8fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:43.277354 24292 consensus_queue.cc:1035] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc05da817a8f4da9ad81d43a3c5d3d53" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:43.277781 24268 consensus_queue.cc:1035] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "402a1a36a7d546eca6c3656616d74784" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:43.282938 24313 mvcc.cc:205] Tried to move back new op lower bound from 7333959038054137856 to 7333959036956315648. Current Snapshot: MvccSnapshot[applied={T|T < 7333959038054137856}]
I20260927 13:55:43.314579 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.307983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:50779 (local address 127.22.26.193:44417)
0927 13:55:43.308212 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:43.308219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:43.308559 (+ 340us) server_negotiation.cc:407] Connection header received
0927 13:55:43.308714 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:43.308719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:43.308814 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:43.309715 (+ 901us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:43.309877 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.311061 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.312497 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.313165 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.313302 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.313826 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:43.313962 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:43.314057 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:43.314190 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 13:55:43.314274 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 13:55:43.315276 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.307934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:44417 (local address 127.22.26.194:50779)
0927 13:55:43.308467 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:43.308483 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:43.308594 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:43.308933 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:43.308936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:43.308945 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:55:43.309286 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:43.309296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.311204 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.311208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:43.312354 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:43.312367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.313371 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.313378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:43.313556 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.313636 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:55:43.314055 (+ 419us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:55:43.314066 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 13:55:43.314155 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 13:55:43.314599 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":289,"threads_started":1}
I20260927 13:55:43.350055 23755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:55:43.350324 23755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:55:43.353648 23755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6304f134852d4b819ff319e59998e003: Sending DeleteTablet for 3 replicas of tablet 14435933c2274113b07899c35c0c1d3c
I20260927 13:55:43.354869 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 14435933c2274113b07899c35c0c1d3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:43 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:44768
I20260927 13:55:43.355326 24171 tablet_service.cc:1514] Processing DeleteTablet for tablet 14435933c2274113b07899c35c0c1d3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:43 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:40384
I20260927 13:55:43.356058 23897 tablet_service.cc:1514] Processing DeleteTablet for tablet 14435933c2274113b07899c35c0c1d3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:43 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:54512
I20260927 13:55:43.367471 24334 tablet_replica.cc:330] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53: stopping tablet replica
I20260927 13:55:43.367710 24334 raft_consensus.cc:2229] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:55:43.367977 24334 raft_consensus.cc:2258] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:43.369105 24334 ts_tablet_manager.cc:1905] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:43.377106 24334 ts_tablet_manager.cc:1918] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:55:43.377250 24334 log.cc:1196] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/14435933c2274113b07899c35c0c1d3c
I20260927 13:55:43.377759 24334 ts_tablet_manager.cc:1939] T 14435933c2274113b07899c35c0c1d3c P fc05da817a8f4da9ad81d43a3c5d3d53: Deleting consensus metadata
I20260927 13:55:43.378944 23743 catalog_manager.cc:4841] TS fc05da817a8f4da9ad81d43a3c5d3d53 (127.22.26.195:33931): tablet 14435933c2274113b07899c35c0c1d3c (table test-table [id=d18f0cd6cd914d03a82b13acff4da318]) successfully deleted
I20260927 13:55:43.395313 24335 tablet_replica.cc:330] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc: stopping tablet replica
I20260927 13:55:43.396564 24333 tablet_replica.cc:330] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784: stopping tablet replica
I20260927 13:55:43.396893 24335 raft_consensus.cc:2229] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:55:43.397665 24335 raft_consensus.cc:2258] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:43.397975 24333 raft_consensus.cc:2229] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:55:43.398296 24333 raft_consensus.cc:2258] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:43.399703 24333 ts_tablet_manager.cc:1905] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:43.400697 24335 ts_tablet_manager.cc:1905] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 13:55:43 dist-test-slave-gkks krb5kdc[23712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517342, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:55:43.431432 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.394537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.22.26.193:44417)
0927 13:55:43.394719 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:43.394725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:43.411697 (+ 16972us) server_negotiation.cc:407] Connection header received
0927 13:55:43.412200 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:43.412206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:43.412292 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:43.412365 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:43.414439 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.415614 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.417523 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.418172 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.418266 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.423630 (+ 5364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:43.423645 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:43.423665 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:43.423705 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:43.427765 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:43.428713 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:43.428723 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:43.428726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:43.428903 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:43.429361 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:43.429367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:43.429370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:43.429885 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:43.430187 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:43.430800 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 13:55:43.431043 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:55:43.432137 24333 ts_tablet_manager.cc:1918] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:55:43.432298 24333 log.cc:1196] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/14435933c2274113b07899c35c0c1d3c
I20260927 13:55:43.432806 24333 ts_tablet_manager.cc:1939] T 14435933c2274113b07899c35c0c1d3c P 402a1a36a7d546eca6c3656616d74784: Deleting consensus metadata
W20260927 13:55:43.433492 23959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54522
I20260927 13:55:43.434141 24335 ts_tablet_manager.cc:1918] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:55:43.434268 24335 log.cc:1196] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/14435933c2274113b07899c35c0c1d3c
I20260927 13:55:43.434273 23744 catalog_manager.cc:4841] TS 402a1a36a7d546eca6c3656616d74784 (127.22.26.194:40193): tablet 14435933c2274113b07899c35c0c1d3c (table test-table [id=d18f0cd6cd914d03a82b13acff4da318]) successfully deleted
I20260927 13:55:43.434746 24335 ts_tablet_manager.cc:1939] T 14435933c2274113b07899c35c0c1d3c P bbb135d70fdd4a1db14249aaf892d8fc: Deleting consensus metadata
I20260927 13:55:43.436757 23742 catalog_manager.cc:4841] TS bbb135d70fdd4a1db14249aaf892d8fc (127.22.26.193:44417): tablet 14435933c2274113b07899c35c0c1d3c (table test-table [id=d18f0cd6cd914d03a82b13acff4da318]) successfully deleted
I20260927 13:55:43.454414 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.438312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.22.26.254:45575)
0927 13:55:43.438516 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:43.438524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:43.438925 (+ 401us) server_negotiation.cc:407] Connection header received
0927 13:55:43.439230 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:43.439236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:43.439315 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:43.439384 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:43.441155 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.442276 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.444143 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.444776 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.444892 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.447868 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:43.447875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:43.447881 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:43.447918 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:43.451636 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:43.452398 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:43.452404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:43.452408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:43.452541 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:43.452936 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:43.452941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:43.452945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:43.453273 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:43.453524 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:43.453836 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 13:55:43.454070 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 13:55:43.455122 23755 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46828
I20260927 13:55:43.457420 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 23828
I20260927 13:55:43.475215 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.459886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.22.26.193:44417)
0927 13:55:43.460060 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:43.460066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:43.460088 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:55:43.460139 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:43.460144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:43.460232 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:43.460326 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:43.461735 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.463014 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.464892 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.465569 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.465689 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.468422 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:43.468428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:43.468436 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:43.468473 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:43.472145 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:43.472984 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:43.472989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:43.472993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:43.473140 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:43.473605 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:43.473616 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:43.473625 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:43.473985 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:43.474250 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:43.474615 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 13:55:43.474837 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 13:55:43.476104 23959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54530
W20260927 13:55:43.476730 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 23828: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:55:43.476917 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 23828
I20260927 13:55:43.496207 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 23968
Sep 27 13:55:43 dist-test-slave-gkks krb5kdc[23712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517342, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:55:43.512780 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.496968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.22.26.194:40193)
0927 13:55:43.497154 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:43.497160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:43.497180 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:55:43.497324 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:43.497336 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:43.497417 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:43.497533 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:43.498667 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.499945 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.501130 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.501866 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.501998 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.506105 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:43.506118 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:43.506136 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:43.506174 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:43.510108 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:43.510712 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:43.510721 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:43.510725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:43.510896 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:43.511294 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:43.511299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:43.511303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:43.511707 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:43.512024 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:43.512169 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 13:55:43.512378 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 13:55:43.513598 24098 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44784
W20260927 13:55:43.514273 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 23968: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:55:43.514472 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 23968
I20260927 13:55:43.533567 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 24107
Sep 27 13:55:43 dist-test-slave-gkks krb5kdc[23712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517342, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:55:43.547387 24258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:43.534269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40400 (local address 127.22.26.195:33931)
0927 13:55:43.534456 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:43.534461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:43.534477 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:55:43.534590 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:43.534594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:43.534663 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:43.534747 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:43.535968 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.536670 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.537821 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:43.538266 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:43.538355 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:43.542252 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:43.542264 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:43.542280 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:43.542311 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:43.545172 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:43.545724 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:43.545732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:43.545735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:43.545860 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:43.546147 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:43.546149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:43.546151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:43.546481 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:43.546715 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:43.546917 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 13:55:43.547124 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 13:55:43.548012 24237 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40400
W20260927 13:55:43.548696 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 24107: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:55:43.548828 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 24107
I20260927 13:55:43.571293 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 23725
I20260927 13:55:43.572425 23797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:43.687779 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 23725
2026-09-27T13:55:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4656 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:55:43 dist-test-slave-gkks krb5kdc[24344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:55:43 dist-test-slave-gkks krb5kdc[24344](info): set up 2 sockets
Sep 27 13:55:43 dist-test-slave-gkks krb5kdc[24344](info): commencing operation
krb5kdc: starting...
W20260927 13:55:44.771673 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:55:44 dist-test-slave-gkks krb5kdc[24344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:55:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:55:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:55:44.948402 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:43017
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:43017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:45.265769 24357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:45.266093 24357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:45.266256 24357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:45.273240 24357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:45.273376 24357 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 13:55:45.273470 24357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:45.273586 24357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:45.273720 24357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:45.273828 24357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:45.283420 24357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:43017
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:43017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:45.284751 24357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:45.286088 24357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:45.297557 24365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:45.297601 24362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:45.297636 24363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:45.298252 24357 server_base.cc:1034] running on GCE node
I20260927 13:55:45.298993 24357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:45.300316 24357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:45.301551 24357 hybrid_clock.cc:648] HybridClock initialized: now 1790517345301483 us; error 34 us; skew 500 ppm
I20260927 13:55:45.301946 24357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:45 dist-test-slave-gkks krb5kdc[24344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:45.305120 24357 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:55:45.306440 24357 webserver.cc:460] Webserver started at http://127.22.26.254:39913/ using document root <none> and password file <none>
I20260927 13:55:45.307118 24357 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:45.307221 24357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:45.307487 24357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:45.310331 24357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "b223d7d3c86a4d8cab513a963f1e1923"
format_stamp: "Formatted at 2026-09-27 13:55:45 on dist-test-slave-gkks"
server_key: "09c04c2ecce8e5a68b8308433c792569"
server_key_iv: "215ae9c8a57de586029051709793bb86"
server_key_version: "encryptionkey@0"
I20260927 13:55:45.311018 24357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "b223d7d3c86a4d8cab513a963f1e1923"
format_stamp: "Formatted at 2026-09-27 13:55:45 on dist-test-slave-gkks"
server_key: "09c04c2ecce8e5a68b8308433c792569"
server_key_iv: "215ae9c8a57de586029051709793bb86"
server_key_version: "encryptionkey@0"
I20260927 13:55:45.316154 24357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:55:45.319401 24372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:45.320636 24357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:55:45.320806 24357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "b223d7d3c86a4d8cab513a963f1e1923"
format_stamp: "Formatted at 2026-09-27 13:55:45 on dist-test-slave-gkks"
server_key: "09c04c2ecce8e5a68b8308433c792569"
server_key_iv: "215ae9c8a57de586029051709793bb86"
server_key_version: "encryptionkey@0"
I20260927 13:55:45.321012 24357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:45.339249 24357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:45.342991 24357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:45.343273 24357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:45.363325 24357 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:43017
I20260927 13:55:45.363334 24433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:43017 every 8 connection(s)
I20260927 13:55:45.364713 24357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:55:45.368568 24434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:45.375319 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 24357
I20260927 13:55:45.375551 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:55:45.376446 22635 external_mini_cluster.cc:1428] Setting key 23ea6604e6c2cf8ca1a9226916530f43
I20260927 13:55:45.380189 24434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923: Bootstrap starting.
I20260927 13:55:45.384037 24434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:45.385164 24434 log.cc:824] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:45.388202 24434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923: No bootstrap required, opened a new log
Sep 27 13:55:45 dist-test-slave-gkks krb5kdc[24344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:45.396948 24434 raft_consensus.cc:348] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b223d7d3c86a4d8cab513a963f1e1923" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43017 } }
I20260927 13:55:45.397365 24434 raft_consensus.cc:374] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:45.397490 24434 raft_consensus.cc:729] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b223d7d3c86a4d8cab513a963f1e1923, State: Initialized, Role: FOLLOWER
I20260927 13:55:45.398241 24434 consensus_queue.cc:260] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b223d7d3c86a4d8cab513a963f1e1923" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43017 } }
I20260927 13:55:45.398564 24434 raft_consensus.cc:388] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:55:45.398727 24434 raft_consensus.cc:482] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:55:45.398936 24434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:45.399919 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:45.378595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.22.26.254:43017)
0927 13:55:45.379395 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:45.379406 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:45.379445 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:55:45.381054 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:45.381075 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:45.381452 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:45.381794 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:45.383730 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:45.385043 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:45.386730 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:45.387424 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:45.387549 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:45.392481 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:45.392488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:45.392501 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:45.392537 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:45.395621 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:45.396315 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:45.396321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:45.396327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:45.396490 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:45.396855 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:45.396861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:45.396865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:45.397433 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:45.397736 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:45.398216 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 13:55:45.398777 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":381,"threads_started":1}
I20260927 13:55:45.401808 24434 raft_consensus.cc:504] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b223d7d3c86a4d8cab513a963f1e1923" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43017 } }
I20260927 13:55:45.402436 24434 leader_election.cc:302] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b223d7d3c86a4d8cab513a963f1e1923; no voters:
I20260927 13:55:45.403175 24434 leader_election.cc:288] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:55:45.403364 24439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:45.404083 24439 raft_consensus.cc:686] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [term 1 LEADER]: Becoming Leader. State: Replica: b223d7d3c86a4d8cab513a963f1e1923, State: Running, Role: LEADER
I20260927 13:55:45.404558 24439 consensus_queue.cc:237] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b223d7d3c86a4d8cab513a963f1e1923" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43017 } }
I20260927 13:55:45.405313 24434 sys_catalog.cc:564] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:45.407339 24440 sys_catalog.cc:455] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b223d7d3c86a4d8cab513a963f1e1923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b223d7d3c86a4d8cab513a963f1e1923" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43017 } } }
I20260927 13:55:45.407758 24440 sys_catalog.cc:458] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:45.407368 24441 sys_catalog.cc:455] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b223d7d3c86a4d8cab513a963f1e1923. Latest consensus state: current_term: 1 leader_uuid: "b223d7d3c86a4d8cab513a963f1e1923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b223d7d3c86a4d8cab513a963f1e1923" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43017 } } }
I20260927 13:55:45.408159 24441 sys_catalog.cc:458] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:45.411544 24444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:55:45.417516 24444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:55:45.431279 24444 catalog_manager.cc:1347] Generated new cluster ID: 0d540746735348eebb4aa534eb1c2e71
I20260927 13:55:45.431392 24444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:55:45.451224 24444 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:55:45.452514 24444 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:55:45.465826 24444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923: Generated new TSK 0
I20260927 13:55:45.466591 24444 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:55:45.518086 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:43017
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:45.863234 24460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:45.863551 24460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:45.863721 24460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:45.870949 24460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:45.871153 24460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:45.871323 24460 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 13:55:45.871418 24460 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:55:45.871542 24460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:55:45.881181 24460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.26.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:45.882483 24460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:45.883818 24460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:45.896703 24465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:45.896746 24466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:45.896780 24468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:45.897583 24460 server_base.cc:1034] running on GCE node
I20260927 13:55:45.898182 24460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:45.899183 24460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:45.900411 24460 hybrid_clock.cc:648] HybridClock initialized: now 1790517345900358 us; error 39 us; skew 500 ppm
I20260927 13:55:45.900725 24460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:45 dist-test-slave-gkks krb5kdc[24344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:45.903815 24460 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:55:45.905223 24460 webserver.cc:460] Webserver started at http://127.22.26.193:41253/ using document root <none> and password file <none>
I20260927 13:55:45.906000 24460 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:45.906100 24460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:45.906469 24460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:45.909515 24460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "5b22a74807ab439ea1954c5528293034"
format_stamp: "Formatted at 2026-09-27 13:55:45 on dist-test-slave-gkks"
server_key: "5b93557d96a746d3aa6aa4c1648d5a0b"
server_key_iv: "228f70093ef812fb35511df8afd9be34"
server_key_version: "encryptionkey@0"
I20260927 13:55:45.910328 24460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "5b22a74807ab439ea1954c5528293034"
format_stamp: "Formatted at 2026-09-27 13:55:45 on dist-test-slave-gkks"
server_key: "5b93557d96a746d3aa6aa4c1648d5a0b"
server_key_iv: "228f70093ef812fb35511df8afd9be34"
server_key_version: "encryptionkey@0"
I20260927 13:55:45.915719 24460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 13:55:45.918875 24475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:45.920168 24460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:55:45.920303 24460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "5b22a74807ab439ea1954c5528293034"
format_stamp: "Formatted at 2026-09-27 13:55:45 on dist-test-slave-gkks"
server_key: "5b93557d96a746d3aa6aa4c1648d5a0b"
server_key_iv: "228f70093ef812fb35511df8afd9be34"
server_key_version: "encryptionkey@0"
I20260927 13:55:45.920459 24460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:45.938514 24460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:45.942253 24460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:45.942490 24460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:45.945029 24460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:45.945122 24460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:45.945264 24460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:45.945355 24460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:55:45 dist-test-slave-gkks krb5kdc[24344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:45.970376 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:45.945479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:59123 (local address 127.22.26.254:43017)
0927 13:55:45.946205 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:45.946212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:45.947264 (+ 1052us) server_negotiation.cc:407] Connection header received
0927 13:55:45.950278 (+ 3014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:45.950287 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:45.950398 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:45.950530 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:45.953100 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:45.954016 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:45.957295 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:45.957867 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:45.958021 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:45.963747 (+ 5726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:45.963755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:45.963759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:45.963804 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:45.966647 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:45.967437 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:45.967442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:45.967446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:45.967624 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:45.968145 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:45.968154 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:45.968160 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:45.968649 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:45.968845 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:45.969506 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 13:55:45.969806 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":479,"threads_started":1}
I20260927 13:55:45.970932 24485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:45.946130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43017 (local address 127.22.26.193:59123)
0927 13:55:45.947064 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:45.947145 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:45.949931 (+ 2786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:45.950851 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:45.950867 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:45.951839 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:45.952880 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:45.952918 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:45.954184 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:45.954197 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:45.957059 (+ 2862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:45.957077 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:45.958147 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:45.958158 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:45.958523 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:45.959492 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:45.959526 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:45.963446 (+ 3920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:45.966789 (+ 3343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:45.966797 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:45.966812 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:45.967293 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:45.967779 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:45.967784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:45.967789 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:45.967996 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:45.968791 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:45.968797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:45.969135 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 13:55:45.969355 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 13:55:45.969730 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":378,"threads_started":1}
I20260927 13:55:45.998863 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:45.982817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:34219 (local address 127.22.26.254:43017)
0927 13:55:45.982977 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:45.982983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:45.984100 (+ 1117us) server_negotiation.cc:407] Connection header received
0927 13:55:45.984233 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:45.984241 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:45.984350 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:45.984488 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:45.985445 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:45.986858 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:45.988871 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:45.989604 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:45.989744 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:45.992692 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:45.992698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:45.992703 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:45.992741 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:45.995801 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:45.996880 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:45.996888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:45.996893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:45.997057 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:45.997433 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:45.997438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:45.997441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:45.997717 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:45.997881 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:45.998273 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 13:55:45.998516 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:55:45.998980 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:45.982148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43017 (local address 127.22.26.193:34219)
0927 13:55:45.983981 (+ 1833us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:45.983998 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:45.984121 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:45.984498 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:45.984501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:45.984987 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:45.985293 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:45.985303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:45.987040 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:45.987047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:45.988646 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:45.988670 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:45.989785 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:45.989791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:45.989955 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:45.990713 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:45.990749 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:45.992349 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:45.995948 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:45.995957 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:45.995964 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:45.996722 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:45.997174 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:45.997181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:45.997184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:45.997316 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:45.997835 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:45.997850 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:45.998037 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 13:55:45.998169 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 13:55:45.998515 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1708,"thread_start_us":1487,"threads_started":1}
I20260927 13:55:46.004192 24460 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:40255
I20260927 13:55:46.004230 24595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:40255 every 8 connection(s)
I20260927 13:55:46.005407 24460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:55:46.007442 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 24460
I20260927 13:55:46.007504 24596 heartbeater.cc:347] Connected to a master server at 127.22.26.254:43017
I20260927 13:55:46.007694 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:55:46.007741 24596 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:46.008365 22635 external_mini_cluster.cc:1428] Setting key 71b97f57bc8d6cf980408eeb4ea77021
I20260927 13:55:46.008389 24596 heartbeater.cc:511] Master 127.22.26.254:43017 requested a full tablet report, sending...
I20260927 13:55:46.010511 24388 ts_manager.cc:194] Registered new tserver with Master: 5b22a74807ab439ea1954c5528293034 (127.22.26.193:40255)
I20260927 13:55:46.012069 24388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:59123
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:55:46.048406 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:43017
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:46.357923 24600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:46.358280 24600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:46.358460 24600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:46.365806 24600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:46.365947 24600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:46.366084 24600 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 13:55:46.366175 24600 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:55:46.366304 24600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:55:46.376464 24600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.26.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:46.377677 24600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:46.378943 24600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:46.391631 24608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:46.391608 24605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:46.391693 24606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:46.392771 24600 server_base.cc:1034] running on GCE node
I20260927 13:55:46.393338 24600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:46.394181 24600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:46.395380 24600 hybrid_clock.cc:648] HybridClock initialized: now 1790517346395331 us; error 37 us; skew 500 ppm
I20260927 13:55:46.395685 24600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:46 dist-test-slave-gkks krb5kdc[24344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517346, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:46.398641 24600 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:55:46.400019 24600 webserver.cc:460] Webserver started at http://127.22.26.194:33047/ using document root <none> and password file <none>
I20260927 13:55:46.400616 24600 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:46.400713 24600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:46.400957 24600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:46.403800 24600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "65f7a8c213d54ff8a7768f887c1be0a0"
format_stamp: "Formatted at 2026-09-27 13:55:46 on dist-test-slave-gkks"
server_key: "e91f8df3bb15ee297f0e6aba71c7d76d"
server_key_iv: "599451fbb6f3d0c160232ae68954ad61"
server_key_version: "encryptionkey@0"
I20260927 13:55:46.404505 24600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "65f7a8c213d54ff8a7768f887c1be0a0"
format_stamp: "Formatted at 2026-09-27 13:55:46 on dist-test-slave-gkks"
server_key: "e91f8df3bb15ee297f0e6aba71c7d76d"
server_key_iv: "599451fbb6f3d0c160232ae68954ad61"
server_key_version: "encryptionkey@0"
I20260927 13:55:46.409511 24600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:55:46.412976 24615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:46.414305 24600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:55:46.414454 24600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "65f7a8c213d54ff8a7768f887c1be0a0"
format_stamp: "Formatted at 2026-09-27 13:55:46 on dist-test-slave-gkks"
server_key: "e91f8df3bb15ee297f0e6aba71c7d76d"
server_key_iv: "599451fbb6f3d0c160232ae68954ad61"
server_key_version: "encryptionkey@0"
I20260927 13:55:46.414692 24600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:46.436643 24600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:46.441984 24600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:46.442261 24600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:46.444988 24600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:46.445104 24600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:46.445228 24600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:46.445319 24600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:55:46 dist-test-slave-gkks krb5kdc[24344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517346, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:46.468189 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:46.445429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:33231 (local address 127.22.26.254:43017)
0927 13:55:46.445645 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:46.445649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:46.446980 (+ 1331us) server_negotiation.cc:407] Connection header received
0927 13:55:46.449107 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:46.449115 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:46.449220 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:46.449343 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:46.451403 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.452524 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.455783 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.456350 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.456482 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:46.461640 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:46.461645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:46.461648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:46.461681 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:46.464782 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:46.465805 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:46.465811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:46.465814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:46.465980 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:46.466423 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:46.466428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:46.466432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:46.466708 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:46.466869 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:46.467618 (+ 749us) server_negotiation.cc:299] Negotiation successful
0927 13:55:46.467845 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:55:46.469586 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:46.446093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43017 (local address 127.22.26.194:33231)
0927 13:55:46.446827 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:46.446868 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:46.448695 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:46.449539 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:46.449552 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:46.450279 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:46.451209 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:46.451241 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.452706 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.452712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:46.455602 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:46.455626 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.456798 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.456812 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:46.457112 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:46.457875 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:46.457905 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:46.461392 (+ 3487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:46.464922 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:46.464930 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:46.464945 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:46.465590 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:46.466096 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:46.466102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:46.466106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:46.466304 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:46.466844 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:46.466851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:46.467254 (+ 403us) client_negotiation.cc:769] Sending connection context
0927 13:55:46.467530 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 13:55:46.468006 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":299,"threads_started":1}
I20260927 13:55:46.488858 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:46.475279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:32783 (local address 127.22.26.254:43017)
0927 13:55:46.475420 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:46.475424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:46.476098 (+ 674us) server_negotiation.cc:407] Connection header received
0927 13:55:46.476285 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:46.476288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:46.476343 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:46.476436 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:46.477303 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.477970 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.480131 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.480532 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.480656 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:46.482344 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:46.482350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:46.482355 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:46.482399 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:46.485636 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:46.486500 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:46.486511 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:46.486514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:46.486631 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:46.487215 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:46.487221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:46.487224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:46.487546 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:46.487719 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:46.488283 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 13:55:46.488505 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 13:55:46.488961 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:46.475377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43017 (local address 127.22.26.194:32783)
0927 13:55:46.476003 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:46.476019 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:46.476155 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:46.476459 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:46.476463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:46.476907 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:46.477170 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:46.477178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.478177 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.478187 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:46.479935 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:46.479956 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.480740 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.480750 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:46.480923 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:46.481508 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:46.481525 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:46.482161 (+ 636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:46.485794 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:46.485799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:46.485803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:46.486330 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:46.486816 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:46.486824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:46.486831 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:46.487036 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:46.487805 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:46.487813 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:46.488016 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 13:55:46.488170 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 13:55:46.488515 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":384,"threads_started":1}
I20260927 13:55:46.501698 24600 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:44001
I20260927 13:55:46.501742 24734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:44001 every 8 connection(s)
I20260927 13:55:46.502907 24600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:55:46.503850 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 24600
I20260927 13:55:46.504033 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:55:46.504601 22635 external_mini_cluster.cc:1428] Setting key c335a7d9913fc403552440905bedfd47
I20260927 13:55:46.505455 24735 heartbeater.cc:347] Connected to a master server at 127.22.26.254:43017
I20260927 13:55:46.505671 24735 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:46.506345 24735 heartbeater.cc:511] Master 127.22.26.254:43017 requested a full tablet report, sending...
I20260927 13:55:46.507812 24388 ts_manager.cc:194] Registered new tserver with Master: 65f7a8c213d54ff8a7768f887c1be0a0 (127.22.26.194:44001)
I20260927 13:55:46.508605 24388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:33231
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:55:46.545985 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:43017
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:46.859998 24739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:46.860288 24739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:46.860481 24739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:46.867851 24739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:46.867964 24739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:46.868156 24739 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 13:55:46.868242 24739 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:55:46.868395 24739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:55:46.878322 24739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.26.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:46.879581 24739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:46.880913 24739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:46.893529 24744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:46.893573 24745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:46.893607 24747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:46.894739 24739 server_base.cc:1034] running on GCE node
I20260927 13:55:46.895505 24739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:46.896400 24739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:46.897652 24739 hybrid_clock.cc:648] HybridClock initialized: now 1790517346897582 us; error 48 us; skew 500 ppm
I20260927 13:55:46.897998 24739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:46 dist-test-slave-gkks krb5kdc[24344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517346, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:46.906529 24739 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:55:46.907982 24739 webserver.cc:460] Webserver started at http://127.22.26.195:39739/ using document root <none> and password file <none>
I20260927 13:55:46.908640 24739 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:46.908740 24739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:46.908998 24739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:46.911908 24739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "df6ff73877284524af9d77757f996281"
format_stamp: "Formatted at 2026-09-27 13:55:46 on dist-test-slave-gkks"
server_key: "d8676eca22c34236c9d62d89a52cf21f"
server_key_iv: "da47d44dff57f4e1ce4e62186502312a"
server_key_version: "encryptionkey@0"
I20260927 13:55:46.912669 24739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "df6ff73877284524af9d77757f996281"
format_stamp: "Formatted at 2026-09-27 13:55:46 on dist-test-slave-gkks"
server_key: "d8676eca22c34236c9d62d89a52cf21f"
server_key_iv: "da47d44dff57f4e1ce4e62186502312a"
server_key_version: "encryptionkey@0"
I20260927 13:55:46.917893 24739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 13:55:46.921463 24754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:46.922768 24739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:55:46.922942 24739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "df6ff73877284524af9d77757f996281"
format_stamp: "Formatted at 2026-09-27 13:55:46 on dist-test-slave-gkks"
server_key: "d8676eca22c34236c9d62d89a52cf21f"
server_key_iv: "da47d44dff57f4e1ce4e62186502312a"
server_key_version: "encryptionkey@0"
I20260927 13:55:46.923152 24739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:46.956670 24739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:46.960769 24739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:46.961079 24739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:46.963788 24739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:46.963924 24739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:46.964052 24739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:46.964164 24739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:55:46 dist-test-slave-gkks krb5kdc[24344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517346, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:46.986323 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:46.964241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:47561 (local address 127.22.26.254:43017)
0927 13:55:46.964420 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:46.964427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:46.965810 (+ 1383us) server_negotiation.cc:407] Connection header received
0927 13:55:46.968054 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:46.968062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:46.968164 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:46.968268 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:46.970434 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.971282 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.973109 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.973671 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.973833 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:46.978820 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:46.978825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:46.978828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:46.978887 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:46.982767 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:46.983747 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:46.983755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:46.983760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:46.983930 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:46.984541 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:46.984548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:46.984553 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:46.984894 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:46.985046 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:46.985747 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 13:55:46.985962 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 13:55:46.987437 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:46.964793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43017 (local address 127.22.26.195:47561)
0927 13:55:46.965630 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:46.965678 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:46.967799 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:46.968544 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:46.968557 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:46.969325 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:46.970254 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:46.970282 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.971439 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.971445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:46.972976 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:46.972987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.974179 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.974189 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:46.974468 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:46.975245 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:46.975279 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:46.978596 (+ 3317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:46.982912 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:46.982923 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:46.982942 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:46.983592 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:46.984047 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:46.984058 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:46.984065 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:46.984389 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:46.985024 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:46.985034 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:46.985358 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 13:55:46.985593 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 13:55:46.986081 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":357,"threads_started":1}
I20260927 13:55:47.010385 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:46.993367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43017 (local address 127.22.26.195:33699)
0927 13:55:46.994067 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:46.994092 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:46.994256 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:46.994571 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:46.994577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:46.995337 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:46.995859 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:46.995878 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.997195 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.997203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:46.998568 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:46.998579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.999711 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.999715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:46.999971 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.000641 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:47.000664 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:47.001712 (+ 1048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:47.007265 (+ 5553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:47.007270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:47.007274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:47.007847 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:47.008282 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:47.008287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:47.008291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:47.008442 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:47.009109 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:47.009115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:47.009335 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 13:55:47.009502 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 13:55:47.009892 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":351,"threads_started":1}
I20260927 13:55:47.011390 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:46.993376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:33699 (local address 127.22.26.254:43017)
0927 13:55:46.993584 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:46.993592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:46.994187 (+ 595us) server_negotiation.cc:407] Connection header received
0927 13:55:46.994378 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:46.994381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:46.994435 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:46.994510 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:46.996054 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.996975 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.998779 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:46.999409 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:46.999530 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.001970 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:47.001978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:47.001984 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:47.002025 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:47.007074 (+ 5049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:47.008021 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:47.008026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:47.008030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:47.008162 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:47.008565 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:47.008570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:47.008574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:47.008905 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:47.009046 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.010811 (+ 1765us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.010973 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:55:47.014832 24596 heartbeater.cc:503] Master 127.22.26.254:43017 was elected leader, sending a full tablet report...
I20260927 13:55:47.018536 24739 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:45049
I20260927 13:55:47.018579 24873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:45049 every 8 connection(s)
I20260927 13:55:47.019760 24739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:55:47.021550 24874 heartbeater.cc:347] Connected to a master server at 127.22.26.254:43017
I20260927 13:55:47.021754 24874 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:47.022348 24874 heartbeater.cc:511] Master 127.22.26.254:43017 requested a full tablet report, sending...
I20260927 13:55:47.022619 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 24739
I20260927 13:55:47.022794 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:55:47.023289 22635 external_mini_cluster.cc:1428] Setting key f24d44e008e9681ce3fc07a38f06d835
I20260927 13:55:47.023685 24388 ts_manager.cc:194] Registered new tserver with Master: df6ff73877284524af9d77757f996281 (127.22.26.195:45049)
I20260927 13:55:47.024739 24388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:47561
I20260927 13:55:47.026245 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 13:55:47 dist-test-slave-gkks krb5kdc[24344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:55:47 dist-test-slave-gkks krb5kdc[24344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:47.070123 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.057426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.22.26.254:43017)
0927 13:55:47.057657 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.057664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.057849 (+ 185us) server_negotiation.cc:407] Connection header received
0927 13:55:47.057997 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.058001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.058076 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.058161 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:47.059495 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.060174 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.061326 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.061714 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.061805 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.065425 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:47.065429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:47.065433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:47.065463 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:47.067999 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:47.068552 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:47.068556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:47.068558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:47.068666 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:47.068954 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:47.068957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:47.068959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:47.069190 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:47.069369 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.069601 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.069758 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 13:55:47.074168 24388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:55:47.080904 24388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:55:47.121572 24888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.092720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:44001 (local address 127.0.0.1:52716)
0927 13:55:47.113548 (+ 20828us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:47.113570 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:47.113750 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:47.114858 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:47.114862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:47.114891 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:47.115394 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.115409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.116888 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.116896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.118745 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.118766 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.120661 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.120667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.120852 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.120901 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 13:55:47.121045 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 13:55:47.121235 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20659,"thread_start_us":20488,"threads_started":1}
I20260927 13:55:47.121568 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.093179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.22.26.194:44001)
0927 13:55:47.114349 (+ 21170us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.114358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.114384 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:55:47.114474 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.114481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.114680 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.114871 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:47.115599 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.116765 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.118928 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.120471 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.120612 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.120826 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.121101 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.121270 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20975,"thread_start_us":21854,"threads_started":1}
I20260927 13:55:47.122537 24890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.093027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40255 (local address 127.0.0.1:56360)
0927 13:55:47.113865 (+ 20838us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:47.113880 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:47.114023 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:47.114741 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:47.114748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:47.114778 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:47.115368 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.115383 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.117512 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.117517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.119707 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.119726 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.121762 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.121771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.121928 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.121977 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 13:55:47.122060 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 13:55:47.122174 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20711,"thread_start_us":20560,"threads_started":1}
I20260927 13:55:47.122886 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.092351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:45049 (local address 127.0.0.1:32940)
0927 13:55:47.108721 (+ 16370us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:47.108762 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:47.108950 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:47.115509 (+ 6559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:47.115517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:47.115547 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:47.115989 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.116002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.118140 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.118146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.120332 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.120343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.122100 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.122106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.122292 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.122330 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 13:55:47.122432 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 13:55:47.122551 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15648,"thread_start_us":15518,"threads_started":1}
I20260927 13:55:47.123005 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.092781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.22.26.195:45049)
0927 13:55:47.114651 (+ 21870us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.114661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.114691 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:55:47.114776 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.114793 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.115015 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.115214 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:47.116398 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.117996 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.120474 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.121897 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.122039 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.122277 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.122472 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.122618 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21681,"thread_start_us":21492,"threads_started":1}
I20260927 13:55:47.123416 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.094028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.22.26.193:40255)
0927 13:55:47.112989 (+ 18961us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.113001 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.113984 (+ 983us) server_negotiation.cc:407] Connection header received
0927 13:55:47.114170 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.114181 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.114491 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.115605 (+ 1114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:47.115819 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.117354 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.119892 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.121264 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.121403 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.121593 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.122899 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.123117 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18746,"thread_start_us":17273,"threads_started":1}
I20260927 13:55:47.126200 24665 tablet_service.cc:1467] Processing CreateTablet for tablet 96781f9662b94c86812d2ddb95ae1646 (DEFAULT_TABLE table=test-table [id=7e4b3862ddcb4f44b44db3e637131ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:47.126413 24526 tablet_service.cc:1467] Processing CreateTablet for tablet 96781f9662b94c86812d2ddb95ae1646 (DEFAULT_TABLE table=test-table [id=7e4b3862ddcb4f44b44db3e637131ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:47.127548 24665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96781f9662b94c86812d2ddb95ae1646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:47.127679 24526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96781f9662b94c86812d2ddb95ae1646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:47.129544 24804 tablet_service.cc:1467] Processing CreateTablet for tablet 96781f9662b94c86812d2ddb95ae1646 (DEFAULT_TABLE table=test-table [id=7e4b3862ddcb4f44b44db3e637131ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:47.130818 24804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96781f9662b94c86812d2ddb95ae1646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:47.138175 24893 tablet_bootstrap.cc:492] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034: Bootstrap starting.
I20260927 13:55:47.141399 24893 tablet_bootstrap.cc:654] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:47.141412 24894 tablet_bootstrap.cc:492] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0: Bootstrap starting.
I20260927 13:55:47.142776 24893 log.cc:824] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:47.144587 24894 tablet_bootstrap.cc:654] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:47.145470 24896 tablet_bootstrap.cc:492] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281: Bootstrap starting.
I20260927 13:55:47.145828 24893 tablet_bootstrap.cc:492] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034: No bootstrap required, opened a new log
I20260927 13:55:47.145774 24894 log.cc:824] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:47.146086 24893 ts_tablet_manager.cc:1397] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 13:55:47.148662 24896 tablet_bootstrap.cc:654] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:47.149624 24894 tablet_bootstrap.cc:492] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0: No bootstrap required, opened a new log
I20260927 13:55:47.149780 24896 log.cc:824] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:47.149893 24894 ts_tablet_manager.cc:1397] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 13:55:47.152086 24896 tablet_bootstrap.cc:492] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281: No bootstrap required, opened a new log
I20260927 13:55:47.152326 24896 ts_tablet_manager.cc:1397] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 13:55:47.156435 24893 raft_consensus.cc:348] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } }
I20260927 13:55:47.156845 24893 raft_consensus.cc:374] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:47.156939 24893 raft_consensus.cc:729] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b22a74807ab439ea1954c5528293034, State: Initialized, Role: FOLLOWER
I20260927 13:55:47.157543 24893 consensus_queue.cc:260] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } }
I20260927 13:55:47.159302 24893 ts_tablet_manager.cc:1428] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260927 13:55:47.160127 24896 raft_consensus.cc:348] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } }
I20260927 13:55:47.160454 24896 raft_consensus.cc:374] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:47.160555 24896 raft_consensus.cc:729] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6ff73877284524af9d77757f996281, State: Initialized, Role: FOLLOWER
I20260927 13:55:47.160347 24894 raft_consensus.cc:348] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } }
I20260927 13:55:47.160718 24894 raft_consensus.cc:374] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:47.160830 24894 raft_consensus.cc:729] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f7a8c213d54ff8a7768f887c1be0a0, State: Initialized, Role: FOLLOWER
I20260927 13:55:47.161103 24896 consensus_queue.cc:260] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } }
I20260927 13:55:47.161372 24894 consensus_queue.cc:260] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } }
I20260927 13:55:47.162163 24874 heartbeater.cc:503] Master 127.22.26.254:43017 was elected leader, sending a full tablet report...
I20260927 13:55:47.163210 24735 heartbeater.cc:503] Master 127.22.26.254:43017 was elected leader, sending a full tablet report...
I20260927 13:55:47.163805 24894 ts_tablet_manager.cc:1428] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260927 13:55:47.165046 24896 ts_tablet_manager.cc:1428] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260927 13:55:47.255173 24736 tablet.cc:2369] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:55:47.257701 24597 tablet.cc:2369] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:55:47.271241 24875 tablet.cc:2369] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:55:47.306188 24899 raft_consensus.cc:482] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:55:47.306607 24899 raft_consensus.cc:504] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } }
I20260927 13:55:47.308527 24899 leader_election.cc:288] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df6ff73877284524af9d77757f996281 (127.22.26.195:45049), 65f7a8c213d54ff8a7768f887c1be0a0 (127.22.26.194:44001)
I20260927 13:55:47.315626 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.308899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:43361 (local address 127.22.26.195:45049)
0927 13:55:47.309150 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.309160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.309832 (+ 672us) server_negotiation.cc:407] Connection header received
0927 13:55:47.310058 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.310066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.310156 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.310225 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:47.310877 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.311733 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.313889 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.314613 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.314806 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.314883 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.315248 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.315367 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:55:47.315780 24903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.308946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:44001 (local address 127.22.26.193:41969)
0927 13:55:47.309854 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:47.309873 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:47.310007 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:47.310385 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:47.310390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:47.310432 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:47.310781 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.310810 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.311910 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.311914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.313675 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.313690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.314840 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.314846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.314992 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.315054 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 13:55:47.315174 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 13:55:47.315316 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":73,"thread_start_us":564,"threads_started":1}
I20260927 13:55:47.315831 24902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.308776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:45049 (local address 127.22.26.193:43361)
0927 13:55:47.309738 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:47.309757 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:47.309876 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:47.310284 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:47.310292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:47.310321 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:47.310723 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.310734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.311853 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.311860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.313703 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.313715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.314877 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.314883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.315053 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.315120 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 13:55:47.315214 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 13:55:47.315347 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":80,"spinlock_wait_cycles":896,"thread_start_us":588,"threads_started":1}
I20260927 13:55:47.316200 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.309003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:41969 (local address 127.22.26.194:44001)
0927 13:55:47.309241 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.309250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.309950 (+ 700us) server_negotiation.cc:407] Connection header received
0927 13:55:47.310154 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.310158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.310247 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.310340 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:47.310955 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.311748 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.313833 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.314611 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.314806 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.314889 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.315216 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.315349 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:55:47.316841 24829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96781f9662b94c86812d2ddb95ae1646" candidate_uuid: "5b22a74807ab439ea1954c5528293034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6ff73877284524af9d77757f996281" is_pre_election: true
I20260927 13:55:47.317060 24690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96781f9662b94c86812d2ddb95ae1646" candidate_uuid: "5b22a74807ab439ea1954c5528293034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" is_pre_election: true
I20260927 13:55:47.317489 24829 raft_consensus.cc:2454] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b22a74807ab439ea1954c5528293034 in term 0.
I20260927 13:55:47.317534 24690 raft_consensus.cc:2454] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b22a74807ab439ea1954c5528293034 in term 0.
I20260927 13:55:47.318233 24479 leader_election.cc:302] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b22a74807ab439ea1954c5528293034, df6ff73877284524af9d77757f996281; no voters:
I20260927 13:55:47.318583 24899 raft_consensus.cc:2781] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:55:47.318728 24899 raft_consensus.cc:482] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:55:47.318840 24899 raft_consensus.cc:3037] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:47.321034 24899 raft_consensus.cc:504] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } }
I20260927 13:55:47.321630 24899 leader_election.cc:288] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [CANDIDATE]: Term 1 election: Requested vote from peers df6ff73877284524af9d77757f996281 (127.22.26.195:45049), 65f7a8c213d54ff8a7768f887c1be0a0 (127.22.26.194:44001)
I20260927 13:55:47.322079 24829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96781f9662b94c86812d2ddb95ae1646" candidate_uuid: "5b22a74807ab439ea1954c5528293034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6ff73877284524af9d77757f996281"
I20260927 13:55:47.322284 24829 raft_consensus.cc:3037] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:47.322213 24690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96781f9662b94c86812d2ddb95ae1646" candidate_uuid: "5b22a74807ab439ea1954c5528293034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65f7a8c213d54ff8a7768f887c1be0a0"
I20260927 13:55:47.322474 24690 raft_consensus.cc:3037] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:47.324931 24829 raft_consensus.cc:2454] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b22a74807ab439ea1954c5528293034 in term 1.
I20260927 13:55:47.324988 24690 raft_consensus.cc:2454] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b22a74807ab439ea1954c5528293034 in term 1.
I20260927 13:55:47.325397 24476 leader_election.cc:302] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b22a74807ab439ea1954c5528293034, 65f7a8c213d54ff8a7768f887c1be0a0; no voters:
I20260927 13:55:47.325752 24899 raft_consensus.cc:2781] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:47.326239 24899 raft_consensus.cc:686] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [term 1 LEADER]: Becoming Leader. State: Replica: 5b22a74807ab439ea1954c5528293034, State: Running, Role: LEADER
I20260927 13:55:47.326815 24899 consensus_queue.cc:237] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } }
I20260927 13:55:47.331590 24389 catalog_manager.cc:5495] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b22a74807ab439ea1954c5528293034 (127.22.26.193). New cstate: current_term: 1 leader_uuid: "5b22a74807ab439ea1954c5528293034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } health_report { overall_health: HEALTHY } } }
Sep 27 13:55:47 dist-test-slave-gkks krb5kdc[24344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:47.398267 24911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.385427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43017 (local address 127.0.0.1:45614)
0927 13:55:47.385964 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:47.385980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:47.386142 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:47.386469 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:47.386474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:47.387205 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:47.387488 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.387503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.388537 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.388541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.389557 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.389569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.390366 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.390370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.390520 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.391212 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:47.391235 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:47.393287 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:47.395972 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:47.395982 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:47.395987 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:47.396415 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:47.396774 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:47.396777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:47.396780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:47.396918 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:47.397470 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:47.397476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:47.397619 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 13:55:47.397769 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 13:55:47.397959 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":280,"threads_started":1}
I20260927 13:55:47.398350 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.385356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.22.26.254:43017)
0927 13:55:47.385519 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.385523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.386066 (+ 543us) server_negotiation.cc:407] Connection header received
0927 13:55:47.386265 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.386270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.386345 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.386443 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:47.387668 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.388423 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.389726 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.390213 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.390306 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.393436 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:47.393441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:47.393444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:47.393472 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:47.395843 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:47.396523 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:47.396526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:47.396529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:47.396650 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:47.397039 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:47.397043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:47.397045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:47.397349 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:47.397470 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.397873 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.398076 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":50}
I20260927 13:55:47.403020 24389 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:45614:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 13:55:47.404032 24389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:55:47.410770 24526 tablet_service.cc:1467] Processing CreateTablet for tablet 12d2aac849b6458488f84684d32bb681 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c99008328164725950f5538fcb76301]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:55:47.410984 24804 tablet_service.cc:1467] Processing CreateTablet for tablet 12d2aac849b6458488f84684d32bb681 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c99008328164725950f5538fcb76301]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:55:47.411003 24665 tablet_service.cc:1467] Processing CreateTablet for tablet 12d2aac849b6458488f84684d32bb681 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c99008328164725950f5538fcb76301]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:55:47.411827 24526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d2aac849b6458488f84684d32bb681. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:47.411805 24665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d2aac849b6458488f84684d32bb681. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:47.411815 24804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d2aac849b6458488f84684d32bb681. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:47.417440 24896 tablet_bootstrap.cc:492] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281: Bootstrap starting.
I20260927 13:55:47.418955 24896 tablet_bootstrap.cc:654] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:47.418996 24894 tablet_bootstrap.cc:492] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0: Bootstrap starting.
I20260927 13:55:47.419535 24893 tablet_bootstrap.cc:492] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034: Bootstrap starting.
I20260927 13:55:47.421087 24894 tablet_bootstrap.cc:654] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:47.421387 24893 tablet_bootstrap.cc:654] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:47.426676 24894 tablet_bootstrap.cc:492] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0: No bootstrap required, opened a new log
I20260927 13:55:47.426862 24894 ts_tablet_manager.cc:1397] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 13:55:47.427119 24893 tablet_bootstrap.cc:492] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034: No bootstrap required, opened a new log
I20260927 13:55:47.427312 24893 ts_tablet_manager.cc:1397] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 13:55:47.427680 24894 raft_consensus.cc:348] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.427944 24894 raft_consensus.cc:374] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:47.428056 24894 raft_consensus.cc:729] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f7a8c213d54ff8a7768f887c1be0a0, State: Initialized, Role: FOLLOWER
I20260927 13:55:47.428292 24894 consensus_queue.cc:260] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.428937 24894 ts_tablet_manager.cc:1428] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 13:55:47.428923 24893 raft_consensus.cc:348] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.429443 24896 tablet_bootstrap.cc:492] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281: No bootstrap required, opened a new log
I20260927 13:55:47.429768 24901 tablet_replica.cc:439] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } }
I20260927 13:55:47.430131 24901 tablet_replica.cc:442] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:47.430114 24896 ts_tablet_manager.cc:1397] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260927 13:55:47.429167 24893 raft_consensus.cc:374] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:47.430542 24893 raft_consensus.cc:729] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b22a74807ab439ea1954c5528293034, State: Initialized, Role: FOLLOWER
I20260927 13:55:47.430776 24893 consensus_queue.cc:260] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.430934 24896 raft_consensus.cc:348] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.431192 24896 raft_consensus.cc:374] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:47.431306 24896 raft_consensus.cc:729] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6ff73877284524af9d77757f996281, State: Initialized, Role: FOLLOWER
I20260927 13:55:47.431412 24893 ts_tablet_manager.cc:1428] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 13:55:47.431643 24899 tablet_replica.cc:439] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } }
I20260927 13:55:47.431890 24899 tablet_replica.cc:442] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:47.432051 24896 consensus_queue.cc:260] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.432567 24896 ts_tablet_manager.cc:1428] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 13:55:47.432684 24900 tablet_replica.cc:439] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } }
I20260927 13:55:47.433050 24900 tablet_replica.cc:442] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:47.557349 24899 raft_consensus.cc:482] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:55:47.557586 24899 raft_consensus.cc:504] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.558367 24899 leader_election.cc:288] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65f7a8c213d54ff8a7768f887c1be0a0 (127.22.26.194:44001), df6ff73877284524af9d77757f996281 (127.22.26.195:45049)
I20260927 13:55:47.558966 24829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d2aac849b6458488f84684d32bb681" candidate_uuid: "5b22a74807ab439ea1954c5528293034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6ff73877284524af9d77757f996281" is_pre_election: true
I20260927 13:55:47.558966 24690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d2aac849b6458488f84684d32bb681" candidate_uuid: "5b22a74807ab439ea1954c5528293034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" is_pre_election: true
I20260927 13:55:47.559237 24829 raft_consensus.cc:2454] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b22a74807ab439ea1954c5528293034 in term 0.
I20260927 13:55:47.559271 24690 raft_consensus.cc:2454] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b22a74807ab439ea1954c5528293034 in term 0.
I20260927 13:55:47.559652 24476 leader_election.cc:302] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b22a74807ab439ea1954c5528293034, 65f7a8c213d54ff8a7768f887c1be0a0; no voters:
I20260927 13:55:47.559921 24899 raft_consensus.cc:2781] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:55:47.560009 24899 raft_consensus.cc:482] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:55:47.560077 24899 raft_consensus.cc:3037] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:47.561995 24899 raft_consensus.cc:504] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.562495 24899 leader_election.cc:288] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [CANDIDATE]: Term 1 election: Requested vote from peers 65f7a8c213d54ff8a7768f887c1be0a0 (127.22.26.194:44001), df6ff73877284524af9d77757f996281 (127.22.26.195:45049)
I20260927 13:55:47.562870 24690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d2aac849b6458488f84684d32bb681" candidate_uuid: "5b22a74807ab439ea1954c5528293034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65f7a8c213d54ff8a7768f887c1be0a0"
I20260927 13:55:47.562906 24829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d2aac849b6458488f84684d32bb681" candidate_uuid: "5b22a74807ab439ea1954c5528293034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6ff73877284524af9d77757f996281"
I20260927 13:55:47.563122 24690 raft_consensus.cc:3037] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:47.563134 24829 raft_consensus.cc:3037] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:47.565965 24829 raft_consensus.cc:2454] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b22a74807ab439ea1954c5528293034 in term 1.
I20260927 13:55:47.565965 24690 raft_consensus.cc:2454] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b22a74807ab439ea1954c5528293034 in term 1.
I20260927 13:55:47.566396 24476 leader_election.cc:302] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5b22a74807ab439ea1954c5528293034, 65f7a8c213d54ff8a7768f887c1be0a0, df6ff73877284524af9d77757f996281; no voters:
I20260927 13:55:47.566699 24899 raft_consensus.cc:2781] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:47.566874 24899 raft_consensus.cc:686] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [term 1 LEADER]: Becoming Leader. State: Replica: 5b22a74807ab439ea1954c5528293034, State: Running, Role: LEADER
I20260927 13:55:47.567180 24899 consensus_queue.cc:237] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.568848 24904 tablet_replica.cc:439] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034: TxnStatusTablet state changed. Reason: New leader 5b22a74807ab439ea1954c5528293034. Latest consensus state: current_term: 1 leader_uuid: "5b22a74807ab439ea1954c5528293034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } }
I20260927 13:55:47.569237 24904 tablet_replica.cc:442] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:55:47.569918 24916 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 13:55:47.570277 24916 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 13:55:47.570272 24386 catalog_manager.cc:5495] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b22a74807ab439ea1954c5528293034 (127.22.26.193). New cstate: current_term: 1 leader_uuid: "5b22a74807ab439ea1954c5528293034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } health_report { overall_health: UNKNOWN } } }
I20260927 13:55:47.589447 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.583555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.22.26.193:40255)
0927 13:55:47.583721 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.583730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.583814 (+ 84us) server_negotiation.cc:407] Connection header received
0927 13:55:47.584105 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.584109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.584179 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.584291 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:47.584720 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.585450 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.587339 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.587757 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.587899 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.588400 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:47.588899 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:47.589041 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.589153 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.589214 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 13:55:47.589593 24911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.583444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40255 (local address 127.0.0.1:56372)
0927 13:55:47.583716 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:47.583737 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:47.583891 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:47.584353 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:47.584359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:47.584372 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:55:47.584575 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.584589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.585578 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.585586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.587201 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.587217 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.587914 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.587921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.588121 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.588191 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:55:47.589022 (+ 831us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:55:47.589043 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 13:55:47.589133 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 13:55:47.589293 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 13:55:47.596542 24829 raft_consensus.cc:1261] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Refusing update from remote peer 5b22a74807ab439ea1954c5528293034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:47.596542 24690 raft_consensus.cc:1261] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Refusing update from remote peer 5b22a74807ab439ea1954c5528293034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:47.597447 24904 consensus_queue.cc:1035] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:47.597790 24899 consensus_queue.cc:1035] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:47.607533 24900 tablet_replica.cc:439] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281: TxnStatusTablet state changed. Reason: New leader 5b22a74807ab439ea1954c5528293034. Latest consensus state: current_term: 1 leader_uuid: "5b22a74807ab439ea1954c5528293034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } }
I20260927 13:55:47.607880 24900 tablet_replica.cc:442] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:47.609553 24901 tablet_replica.cc:439] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0: TxnStatusTablet state changed. Reason: New leader 5b22a74807ab439ea1954c5528293034. Latest consensus state: current_term: 1 leader_uuid: "5b22a74807ab439ea1954c5528293034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } }
I20260927 13:55:47.609853 24901 tablet_replica.cc:442] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:47.612504 24919 mvcc.cc:205] Tried to move back new op lower bound from 7333959055746486272 to 7333959055637696512. Current Snapshot: MvccSnapshot[applied={T|T < 7333959055746486272}]
I20260927 13:55:47.614421 24921 mvcc.cc:205] Tried to move back new op lower bound from 7333959055746486272 to 7333959055637696512. Current Snapshot: MvccSnapshot[applied={T|T < 7333959055746486272}]
I20260927 13:55:47.620242 24904 tablet_replica.cc:439] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b22a74807ab439ea1954c5528293034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } }
I20260927 13:55:47.620577 24904 tablet_replica.cc:442] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:55:47.624125 24924 tablet_replica.cc:439] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b22a74807ab439ea1954c5528293034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } }
I20260927 13:55:47.629740 24924 tablet_replica.cc:442] T 12d2aac849b6458488f84684d32bb681 P 5b22a74807ab439ea1954c5528293034: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:55:47.629607 24900 tablet_replica.cc:439] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b22a74807ab439ea1954c5528293034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } }
I20260927 13:55:47.629863 24900 tablet_replica.cc:442] T 12d2aac849b6458488f84684d32bb681 P df6ff73877284524af9d77757f996281: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:47.631510 24901 tablet_replica.cc:439] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b22a74807ab439ea1954c5528293034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } } }
I20260927 13:55:47.631752 24901 tablet_replica.cc:442] T 12d2aac849b6458488f84684d32bb681 P 65f7a8c213d54ff8a7768f887c1be0a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:55:47.639552 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.633389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.22.26.193:40255)
0927 13:55:47.633572 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.633577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.633989 (+ 412us) server_negotiation.cc:407] Connection header received
0927 13:55:47.634167 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.634170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.634249 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.634326 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:47.634782 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.635581 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.637605 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.638028 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.638120 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.638596 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:47.638726 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:47.638795 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.638969 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.639027 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 13:55:47.656397 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.650400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:44757 (local address 127.22.26.193:40255)
0927 13:55:47.650530 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.650534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.650975 (+ 441us) server_negotiation.cc:407] Connection header received
0927 13:55:47.651172 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.651180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.651250 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.651335 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:47.651880 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.652575 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.654636 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.655216 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.655313 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.655812 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:47.655905 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:47.655971 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.656119 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.656173 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 13:55:47.656924 24932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.650243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40255 (local address 127.22.26.193:44757)
0927 13:55:47.650854 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:47.650874 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:47.651021 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:47.651364 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:47.651369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:47.651384 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:55:47.651726 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.651737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.652703 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.652709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.654506 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.654521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.655389 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.655395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.655566 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.655636 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:55:47.655996 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:55:47.656013 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 13:55:47.656087 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 13:55:47.656240 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":264,"threads_started":1}
I20260927 13:55:47.666810 24690 raft_consensus.cc:1261] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Refusing update from remote peer 5b22a74807ab439ea1954c5528293034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:47.666812 24829 raft_consensus.cc:1261] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Refusing update from remote peer 5b22a74807ab439ea1954c5528293034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:47.667382 24924 consensus_queue.cc:1035] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:47.667734 24904 consensus_queue.cc:1035] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:47.723621 24386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:55:47.723896 24386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:55:47.727648 24386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923: Sending DeleteTablet for 3 replicas of tablet 96781f9662b94c86812d2ddb95ae1646
I20260927 13:55:47.728710 24804 tablet_service.cc:1514] Processing DeleteTablet for tablet 96781f9662b94c86812d2ddb95ae1646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:47 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:32940
I20260927 13:55:47.728789 24665 tablet_service.cc:1514] Processing DeleteTablet for tablet 96781f9662b94c86812d2ddb95ae1646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:47 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:52716
I20260927 13:55:47.729625 24936 tablet_replica.cc:330] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281: stopping tablet replica
I20260927 13:55:47.729862 24525 tablet_service.cc:1514] Processing DeleteTablet for tablet 96781f9662b94c86812d2ddb95ae1646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:47 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:56360
I20260927 13:55:47.730322 24936 raft_consensus.cc:2229] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:55:47.730855 24936 raft_consensus.cc:2258] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:47.733268 24936 ts_tablet_manager.cc:1905] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:47.733633 24389 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:55:47.734303 24389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:55:47.740644 24938 tablet_replica.cc:330] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034: stopping tablet replica
I20260927 13:55:47.741093 24938 raft_consensus.cc:2229] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:55:47.741811 24938 raft_consensus.cc:2258] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:47.742254 24664 tablet_service.cc:1467] Processing CreateTablet for tablet 5a253cc4835045ce807380218fe55d54 (DEFAULT_TABLE table=test-table [id=7a970c94a5c94252ab4fe3f479d3958e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:47.742431 24804 tablet_service.cc:1467] Processing CreateTablet for tablet 5a253cc4835045ce807380218fe55d54 (DEFAULT_TABLE table=test-table [id=7a970c94a5c94252ab4fe3f479d3958e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:47.742758 24664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a253cc4835045ce807380218fe55d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:47.742792 24804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a253cc4835045ce807380218fe55d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:47.743839 24526 tablet_service.cc:1467] Processing CreateTablet for tablet 5a253cc4835045ce807380218fe55d54 (DEFAULT_TABLE table=test-table [id=7a970c94a5c94252ab4fe3f479d3958e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:47.744241 24526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a253cc4835045ce807380218fe55d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:47.749022 24896 tablet_bootstrap.cc:492] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281: Bootstrap starting.
I20260927 13:55:47.749432 24894 tablet_bootstrap.cc:492] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0: Bootstrap starting.
I20260927 13:55:47.750713 24896 tablet_bootstrap.cc:654] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:47.750962 24893 tablet_bootstrap.cc:492] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034: Bootstrap starting.
I20260927 13:55:47.751044 24894 tablet_bootstrap.cc:654] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:47.752559 24893 tablet_bootstrap.cc:654] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:47.758862 24936 ts_tablet_manager.cc:1918] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:55:47.759164 24936 log.cc:1196] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/96781f9662b94c86812d2ddb95ae1646
I20260927 13:55:47.759847 24936 ts_tablet_manager.cc:1939] T 96781f9662b94c86812d2ddb95ae1646 P df6ff73877284524af9d77757f996281: Deleting consensus metadata
I20260927 13:55:47.760917 24894 tablet_bootstrap.cc:492] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0: No bootstrap required, opened a new log
I20260927 13:55:47.761097 24894 ts_tablet_manager.cc:1397] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260927 13:55:47.761888 24894 raft_consensus.cc:348] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.762157 24894 raft_consensus.cc:374] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:47.762261 24894 raft_consensus.cc:729] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f7a8c213d54ff8a7768f887c1be0a0, State: Initialized, Role: FOLLOWER
I20260927 13:55:47.762346 24376 catalog_manager.cc:4841] TS df6ff73877284524af9d77757f996281 (127.22.26.195:45049): tablet 96781f9662b94c86812d2ddb95ae1646 (table test-table [id=7e4b3862ddcb4f44b44db3e637131ffb]) successfully deleted
I20260927 13:55:47.762595 24894 consensus_queue.cc:260] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.763214 24894 ts_tablet_manager.cc:1428] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 13:55:47.764364 24937 tablet_replica.cc:330] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0: stopping tablet replica
I20260927 13:55:47.765192 24896 tablet_bootstrap.cc:492] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281: No bootstrap required, opened a new log
I20260927 13:55:47.765357 24896 ts_tablet_manager.cc:1397] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s
I20260927 13:55:47.766116 24896 raft_consensus.cc:348] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.766371 24896 raft_consensus.cc:374] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:47.766458 24896 raft_consensus.cc:729] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6ff73877284524af9d77757f996281, State: Initialized, Role: FOLLOWER
I20260927 13:55:47.766702 24896 consensus_queue.cc:260] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.767709 24896 ts_tablet_manager.cc:1428] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 13:55:47.769037 24938 ts_tablet_manager.cc:1905] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:47.771641 24937 raft_consensus.cc:2229] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:55:47.772027 24937 raft_consensus.cc:2258] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:47.773728 24893 tablet_bootstrap.cc:492] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034: No bootstrap required, opened a new log
I20260927 13:55:47.773962 24893 ts_tablet_manager.cc:1397] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.007s
I20260927 13:55:47.774226 24937 ts_tablet_manager.cc:1905] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:47.774786 24893 raft_consensus.cc:348] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.775027 24893 raft_consensus.cc:374] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:47.775167 24893 raft_consensus.cc:729] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b22a74807ab439ea1954c5528293034, State: Initialized, Role: FOLLOWER
I20260927 13:55:47.775418 24893 consensus_queue.cc:260] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.776724 24893 ts_tablet_manager.cc:1428] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 13:55:47.780709 24937 ts_tablet_manager.cc:1918] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:55:47.780871 24937 log.cc:1196] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/96781f9662b94c86812d2ddb95ae1646
I20260927 13:55:47.781328 24937 ts_tablet_manager.cc:1939] T 96781f9662b94c86812d2ddb95ae1646 P 65f7a8c213d54ff8a7768f887c1be0a0: Deleting consensus metadata
I20260927 13:55:47.782800 24373 catalog_manager.cc:4841] TS 65f7a8c213d54ff8a7768f887c1be0a0 (127.22.26.194:44001): tablet 96781f9662b94c86812d2ddb95ae1646 (table test-table [id=7e4b3862ddcb4f44b44db3e637131ffb]) successfully deleted
I20260927 13:55:47.786897 24938 ts_tablet_manager.cc:1918] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:55:47.787060 24938 log.cc:1196] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/96781f9662b94c86812d2ddb95ae1646
I20260927 13:55:47.787567 24938 ts_tablet_manager.cc:1939] T 96781f9662b94c86812d2ddb95ae1646 P 5b22a74807ab439ea1954c5528293034: Deleting consensus metadata
I20260927 13:55:47.789135 24375 catalog_manager.cc:4841] TS 5b22a74807ab439ea1954c5528293034 (127.22.26.193:40255): tablet 96781f9662b94c86812d2ddb95ae1646 (table test-table [id=7e4b3862ddcb4f44b44db3e637131ffb]) successfully deleted
I20260927 13:55:47.829028 24901 raft_consensus.cc:482] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:55:47.829277 24901 raft_consensus.cc:504] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.831101 24901 leader_election.cc:288] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b22a74807ab439ea1954c5528293034 (127.22.26.193:40255), df6ff73877284524af9d77757f996281 (127.22.26.195:45049)
I20260927 13:55:47.839203 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.831633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:60973 (local address 127.22.26.193:40255)
0927 13:55:47.831807 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.831814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.832358 (+ 544us) server_negotiation.cc:407] Connection header received
0927 13:55:47.832674 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.832680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.832756 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.832823 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:47.833527 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.834638 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.837285 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.838273 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.838455 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.838536 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.838792 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.838918 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:55:47.839306 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.831354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40255 (local address 127.22.26.194:60973)
0927 13:55:47.832252 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:47.832272 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:47.832471 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:47.833058 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:47.833063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:47.833090 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:47.833325 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.833338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.834833 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.834839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.837122 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.837139 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.838435 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.838442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.838639 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.838663 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 13:55:47.838767 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 13:55:47.838894 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":969,"threads_started":1}
I20260927 13:55:47.840137 24550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a253cc4835045ce807380218fe55d54" candidate_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b22a74807ab439ea1954c5528293034" is_pre_election: true
I20260927 13:55:47.840581 24550 raft_consensus.cc:2454] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65f7a8c213d54ff8a7768f887c1be0a0 in term 0.
I20260927 13:55:47.841176 24618 leader_election.cc:302] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b22a74807ab439ea1954c5528293034, 65f7a8c213d54ff8a7768f887c1be0a0; no voters:
I20260927 13:55:47.841631 24901 raft_consensus.cc:2781] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:55:47.841800 24901 raft_consensus.cc:482] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:55:47.841928 24901 raft_consensus.cc:3037] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:47.842447 24943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.831421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:45049 (local address 127.22.26.194:47877)
0927 13:55:47.835618 (+ 4197us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:47.835635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:47.835782 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:47.836082 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:47.836086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:47.836110 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:47.836460 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.836468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.837557 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.837564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.840060 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.840077 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.841650 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.841657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.841826 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.841859 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:55:47.841994 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 13:55:47.842126 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4084,"thread_start_us":3878,"threads_started":1}
I20260927 13:55:47.843156 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.831898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:47877 (local address 127.22.26.195:45049)
0927 13:55:47.833855 (+ 1957us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.833864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.835717 (+ 1853us) server_negotiation.cc:407] Connection header received
0927 13:55:47.835911 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.835915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.835971 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.836067 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:47.836603 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.837425 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.840244 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.841426 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.841585 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.841665 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.842081 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.842248 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1802,"thread_start_us":1678,"threads_started":1}
I20260927 13:55:47.843750 24828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a253cc4835045ce807380218fe55d54" candidate_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6ff73877284524af9d77757f996281" is_pre_election: true
I20260927 13:55:47.844007 24828 raft_consensus.cc:2454] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65f7a8c213d54ff8a7768f887c1be0a0 in term 0.
I20260927 13:55:47.844890 24901 raft_consensus.cc:504] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.845482 24901 leader_election.cc:288] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [CANDIDATE]: Term 1 election: Requested vote from peers 5b22a74807ab439ea1954c5528293034 (127.22.26.193:40255), df6ff73877284524af9d77757f996281 (127.22.26.195:45049)
I20260927 13:55:47.845865 24550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a253cc4835045ce807380218fe55d54" candidate_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b22a74807ab439ea1954c5528293034"
I20260927 13:55:47.845920 24828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a253cc4835045ce807380218fe55d54" candidate_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6ff73877284524af9d77757f996281"
I20260927 13:55:47.846069 24550 raft_consensus.cc:3037] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:47.846096 24828 raft_consensus.cc:3037] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:47.848244 24550 raft_consensus.cc:2454] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65f7a8c213d54ff8a7768f887c1be0a0 in term 1.
I20260927 13:55:47.848623 24618 leader_election.cc:302] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b22a74807ab439ea1954c5528293034, 65f7a8c213d54ff8a7768f887c1be0a0; no voters:
I20260927 13:55:47.848898 24828 raft_consensus.cc:2454] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65f7a8c213d54ff8a7768f887c1be0a0 in term 1.
I20260927 13:55:47.848919 24901 raft_consensus.cc:2781] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:47.849531 24901 raft_consensus.cc:686] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 65f7a8c213d54ff8a7768f887c1be0a0, State: Running, Role: LEADER
I20260927 13:55:47.849931 24901 consensus_queue.cc:237] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:47.853571 24386 catalog_manager.cc:5495] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65f7a8c213d54ff8a7768f887c1be0a0 (127.22.26.194). New cstate: current_term: 1 leader_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } health_report { overall_health: UNKNOWN } } }
I20260927 13:55:47.896168 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.889771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.22.26.194:44001)
0927 13:55:47.890440 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.890444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.890464 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:55:47.890535 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.890539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.890608 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.890743 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:47.891196 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.891970 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.893848 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.894258 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.894348 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.894861 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:47.895482 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:47.895591 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.895739 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.895802 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":402,"threads_started":1}
I20260927 13:55:47.911680 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.906173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:51083 (local address 127.22.26.193:40255)
0927 13:55:47.906420 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.906428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.906754 (+ 326us) server_negotiation.cc:407] Connection header received
0927 13:55:47.907034 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.907038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.907170 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.907273 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:47.907739 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.908507 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.909909 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.910345 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.910470 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.910888 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:47.911006 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:47.911129 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.911261 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.911363 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 13:55:47.911788 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.906088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40255 (local address 127.22.26.194:51083)
0927 13:55:47.906650 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:47.906665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:47.906857 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:47.907288 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:47.907293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:47.907304 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:55:47.907600 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.907608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.908656 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.908662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.909781 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.909793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.910525 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.910533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.910662 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.910717 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:55:47.911128 (+ 411us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:55:47.911140 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 13:55:47.911234 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 13:55:47.911392 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":305,"threads_started":1}
I20260927 13:55:47.922904 24828 raft_consensus.cc:1261] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Refusing update from remote peer 65f7a8c213d54ff8a7768f887c1be0a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:47.923203 24550 raft_consensus.cc:1261] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Refusing update from remote peer 65f7a8c213d54ff8a7768f887c1be0a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:47.923709 24945 consensus_queue.cc:1035] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:47.924324 24901 consensus_queue.cc:1035] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:47.931238 24920 mvcc.cc:205] Tried to move back new op lower bound from 7333959057085747200 to 7333959056796672000. Current Snapshot: MvccSnapshot[applied={T|T < 7333959057085747200}]
I20260927 13:55:47.956265 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.950350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:39631 (local address 127.22.26.194:44001)
0927 13:55:47.950512 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:47.950516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:47.950532 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:55:47.950679 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:47.950686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:47.950763 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:47.950852 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:47.951316 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.952415 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.954262 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.954700 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.954825 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.955564 (+ 739us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:47.955686 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:47.955796 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:47.955905 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 13:55:47.955976 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 13:55:47.956321 24932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:47.949973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:44001 (local address 127.22.26.193:39631)
0927 13:55:47.950404 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:47.950425 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:47.950535 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:47.950893 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:47.950898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:47.950941 (+ 43us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:55:47.951150 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.951163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.952556 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.952561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.954102 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:47.954116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:47.954957 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:47.954966 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:47.955275 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:47.955344 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:55:47.955797 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:55:47.955811 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 13:55:47.955886 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 13:55:47.956033 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281}
W20260927 13:55:47.957612 24921 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 13:55:49.578051 24386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:55:49.578320 24386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:55:49.582134 24386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923: Sending DeleteTablet for 3 replicas of tablet 5a253cc4835045ce807380218fe55d54
I20260927 13:55:49.583006 24665 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a253cc4835045ce807380218fe55d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:49 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:52716
I20260927 13:55:49.583050 24525 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a253cc4835045ce807380218fe55d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:49 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:56360
I20260927 13:55:49.583112 24804 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a253cc4835045ce807380218fe55d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:49 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:32940
I20260927 13:55:49.583760 24980 tablet_replica.cc:330] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034: stopping tablet replica
I20260927 13:55:49.584070 24980 raft_consensus.cc:2229] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:55:49.584398 24980 raft_consensus.cc:2258] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:49.584972 24981 tablet_replica.cc:330] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281: stopping tablet replica
I20260927 13:55:49.585887 24980 ts_tablet_manager.cc:1905] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:49.592016 24979 tablet_replica.cc:330] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0: stopping tablet replica
I20260927 13:55:49.592011 24981 raft_consensus.cc:2229] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:55:49.592206 24979 raft_consensus.cc:2229] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:55:49.592258 24981 raft_consensus.cc:2258] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:49.592769 24979 raft_consensus.cc:2258] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:49.593199 24981 ts_tablet_manager.cc:1905] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:49.594259 24980 ts_tablet_manager.cc:1918] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:55:49.594398 24980 log.cc:1196] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/5a253cc4835045ce807380218fe55d54
I20260927 13:55:49.594838 24980 ts_tablet_manager.cc:1939] T 5a253cc4835045ce807380218fe55d54 P 5b22a74807ab439ea1954c5528293034: Deleting consensus metadata
I20260927 13:55:49.595954 24375 catalog_manager.cc:4841] TS 5b22a74807ab439ea1954c5528293034 (127.22.26.193:40255): tablet 5a253cc4835045ce807380218fe55d54 (table test-table [id=7a970c94a5c94252ab4fe3f479d3958e]) successfully deleted
I20260927 13:55:49.598551 24981 ts_tablet_manager.cc:1918] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:55:49.598651 24981 log.cc:1196] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/5a253cc4835045ce807380218fe55d54
I20260927 13:55:49.599139 24981 ts_tablet_manager.cc:1939] T 5a253cc4835045ce807380218fe55d54 P df6ff73877284524af9d77757f996281: Deleting consensus metadata
I20260927 13:55:49.599452 24979 ts_tablet_manager.cc:1905] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:49.600135 24376 catalog_manager.cc:4841] TS df6ff73877284524af9d77757f996281 (127.22.26.195:45049): tablet 5a253cc4835045ce807380218fe55d54 (table test-table [id=7a970c94a5c94252ab4fe3f479d3958e]) successfully deleted
I20260927 13:55:49.604040 24979 ts_tablet_manager.cc:1918] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:55:49.604192 24979 log.cc:1196] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/5a253cc4835045ce807380218fe55d54
I20260927 13:55:49.604636 24979 ts_tablet_manager.cc:1939] T 5a253cc4835045ce807380218fe55d54 P 65f7a8c213d54ff8a7768f887c1be0a0: Deleting consensus metadata
I20260927 13:55:49.605769 24373 catalog_manager.cc:4841] TS 65f7a8c213d54ff8a7768f887c1be0a0 (127.22.26.194:44001): tablet 5a253cc4835045ce807380218fe55d54 (table test-table [id=7a970c94a5c94252ab4fe3f479d3958e]) successfully deleted
Sep 27 13:55:49 dist-test-slave-gkks krb5kdc[24344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:55:49 dist-test-slave-gkks krb5kdc[24344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:49.635345 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:49.620613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.22.26.254:43017)
0927 13:55:49.621159 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:49.621163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:49.621232 (+ 69us) server_negotiation.cc:407] Connection header received
0927 13:55:49.621391 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:49.621395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:49.621461 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:49.621552 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:49.622986 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.623962 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.625655 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.626228 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.626353 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:49.630280 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:49.630286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:49.630290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:49.630322 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:49.633038 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:49.633671 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:49.633675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:49.633678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:49.633824 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:49.634165 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:49.634167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:49.634170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:49.634426 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:49.634595 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:49.634856 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 13:55:49.635011 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":299,"threads_started":1}
I20260927 13:55:49.638618 24386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:55:49.639250 24386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:55:49.645447 24525 tablet_service.cc:1467] Processing CreateTablet for tablet 0f5693cd0ec846ec92b20e261f11d48d (DEFAULT_TABLE table=test-table [id=69157363d9164abf87e5c6fc68765e65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:49.645511 24665 tablet_service.cc:1467] Processing CreateTablet for tablet 0f5693cd0ec846ec92b20e261f11d48d (DEFAULT_TABLE table=test-table [id=69157363d9164abf87e5c6fc68765e65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:49.645759 24804 tablet_service.cc:1467] Processing CreateTablet for tablet 0f5693cd0ec846ec92b20e261f11d48d (DEFAULT_TABLE table=test-table [id=69157363d9164abf87e5c6fc68765e65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:49.645900 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f5693cd0ec846ec92b20e261f11d48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:49.645900 24665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f5693cd0ec846ec92b20e261f11d48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:49.646162 24804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f5693cd0ec846ec92b20e261f11d48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:49.651597 24990 tablet_bootstrap.cc:492] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281: Bootstrap starting.
I20260927 13:55:49.652592 24991 tablet_bootstrap.cc:492] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0: Bootstrap starting.
I20260927 13:55:49.653393 24992 tablet_bootstrap.cc:492] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034: Bootstrap starting.
I20260927 13:55:49.654297 24991 tablet_bootstrap.cc:654] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:49.654364 24990 tablet_bootstrap.cc:654] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:49.654739 24992 tablet_bootstrap.cc:654] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:49.656611 24991 tablet_bootstrap.cc:492] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0: No bootstrap required, opened a new log
I20260927 13:55:49.656754 24991 ts_tablet_manager.cc:1397] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 13:55:49.657464 24990 tablet_bootstrap.cc:492] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281: No bootstrap required, opened a new log
I20260927 13:55:49.657500 24992 tablet_bootstrap.cc:492] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034: No bootstrap required, opened a new log
I20260927 13:55:49.657608 24990 ts_tablet_manager.cc:1397] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 13:55:49.657646 24992 ts_tablet_manager.cc:1397] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 13:55:49.657588 24991 raft_consensus.cc:348] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:49.657848 24991 raft_consensus.cc:374] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:49.657943 24991 raft_consensus.cc:729] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f7a8c213d54ff8a7768f887c1be0a0, State: Initialized, Role: FOLLOWER
I20260927 13:55:49.658169 24991 consensus_queue.cc:260] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:49.658362 24990 raft_consensus.cc:348] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:49.658566 24990 raft_consensus.cc:374] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:49.658439 24992 raft_consensus.cc:348] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:49.658653 24990 raft_consensus.cc:729] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6ff73877284524af9d77757f996281, State: Initialized, Role: FOLLOWER
I20260927 13:55:49.658711 24992 raft_consensus.cc:374] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:49.658783 24991 ts_tablet_manager.cc:1428] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 13:55:49.658805 24992 raft_consensus.cc:729] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b22a74807ab439ea1954c5528293034, State: Initialized, Role: FOLLOWER
I20260927 13:55:49.658886 24990 consensus_queue.cc:260] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:49.659024 24992 consensus_queue.cc:260] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:49.659822 24992 ts_tablet_manager.cc:1428] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 13:55:49.660713 24990 ts_tablet_manager.cc:1428] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 13:55:49.875757 24996 raft_consensus.cc:482] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:55:49.875993 24996 raft_consensus.cc:504] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:49.877441 24996 leader_election.cc:288] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b22a74807ab439ea1954c5528293034 (127.22.26.193:40255), 65f7a8c213d54ff8a7768f887c1be0a0 (127.22.26.194:44001)
I20260927 13:55:49.886543 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:49.877899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:45889 (local address 127.22.26.193:40255)
0927 13:55:49.879700 (+ 1801us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:49.879704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:49.879724 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:55:49.879784 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:49.879787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:49.879840 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:49.879927 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:49.880593 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.881709 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.884163 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.885377 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.885544 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:49.885616 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:49.886073 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 13:55:49.886223 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1691,"thread_start_us":1537,"threads_started":1}
I20260927 13:55:49.886546 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:49.877899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:39937 (local address 127.22.26.194:44001)
0927 13:55:49.879436 (+ 1537us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:49.879441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:49.879459 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:55:49.879520 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:49.879523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:49.879581 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:49.879690 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:55:49.880291 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.881496 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.884151 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.885378 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.885544 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:49.885625 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:49.886073 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 13:55:49.886214 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1414,"thread_start_us":1261,"threads_started":1}
I20260927 13:55:49.886650 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:49.877703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40255 (local address 127.22.26.195:45889)
0927 13:55:49.878348 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:49.878375 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:49.878559 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:49.879972 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:49.879977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:49.880012 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:49.880422 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:49.880430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.881834 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.881839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:49.883994 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:49.884003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.885628 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.885635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:49.885881 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:49.885937 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 13:55:49.886027 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 13:55:49.886216 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":140,"thread_start_us":321,"threads_started":1}
I20260927 13:55:49.886675 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:49.877703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:44001 (local address 127.22.26.195:39937)
0927 13:55:49.879116 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:49.879135 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:49.879250 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:49.879710 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:49.879715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:49.879739 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:55:49.880096 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:49.880108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.881628 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.881631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:49.883988 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:49.884003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.885648 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.885655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:49.885875 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:49.885930 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 13:55:49.886027 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 13:55:49.886215 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1242,"mutex_wait_us":215,"thread_start_us":1075,"threads_started":1}
I20260927 13:55:49.887293 24690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f5693cd0ec846ec92b20e261f11d48d" candidate_uuid: "df6ff73877284524af9d77757f996281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" is_pre_election: true
I20260927 13:55:49.887562 24690 raft_consensus.cc:2454] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6ff73877284524af9d77757f996281 in term 0.
I20260927 13:55:49.887547 24550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f5693cd0ec846ec92b20e261f11d48d" candidate_uuid: "df6ff73877284524af9d77757f996281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b22a74807ab439ea1954c5528293034" is_pre_election: true
I20260927 13:55:49.887804 24550 raft_consensus.cc:2454] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6ff73877284524af9d77757f996281 in term 0.
I20260927 13:55:49.888084 24755 leader_election.cc:302] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65f7a8c213d54ff8a7768f887c1be0a0, df6ff73877284524af9d77757f996281; no voters:
I20260927 13:55:49.888520 24996 raft_consensus.cc:2781] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:55:49.888643 24996 raft_consensus.cc:482] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:55:49.888748 24996 raft_consensus.cc:3037] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:49.890867 24996 raft_consensus.cc:504] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:49.891479 24996 leader_election.cc:288] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [CANDIDATE]: Term 1 election: Requested vote from peers 5b22a74807ab439ea1954c5528293034 (127.22.26.193:40255), 65f7a8c213d54ff8a7768f887c1be0a0 (127.22.26.194:44001)
I20260927 13:55:49.891952 24690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f5693cd0ec846ec92b20e261f11d48d" candidate_uuid: "df6ff73877284524af9d77757f996281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65f7a8c213d54ff8a7768f887c1be0a0"
I20260927 13:55:49.892179 24690 raft_consensus.cc:3037] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:49.892096 24550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f5693cd0ec846ec92b20e261f11d48d" candidate_uuid: "df6ff73877284524af9d77757f996281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b22a74807ab439ea1954c5528293034"
I20260927 13:55:49.892315 24550 raft_consensus.cc:3037] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:49.895207 24550 raft_consensus.cc:2454] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6ff73877284524af9d77757f996281 in term 1.
I20260927 13:55:49.895207 24690 raft_consensus.cc:2454] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6ff73877284524af9d77757f996281 in term 1.
I20260927 13:55:49.895668 24757 leader_election.cc:302] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b22a74807ab439ea1954c5528293034, df6ff73877284524af9d77757f996281; no voters:
I20260927 13:55:49.896046 24996 raft_consensus.cc:2781] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:49.896433 24996 raft_consensus.cc:686] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [term 1 LEADER]: Becoming Leader. State: Replica: df6ff73877284524af9d77757f996281, State: Running, Role: LEADER
I20260927 13:55:49.896847 24996 consensus_queue.cc:237] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:49.900010 24388 catalog_manager.cc:5495] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 reported cstate change: term changed from 0 to 1, leader changed from <none> to df6ff73877284524af9d77757f996281 (127.22.26.195). New cstate: current_term: 1 leader_uuid: "df6ff73877284524af9d77757f996281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } health_report { overall_health: HEALTHY } } }
I20260927 13:55:49.959015 25004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:49.953226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.22.26.195:45049)
0927 13:55:49.954000 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:49.954005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:49.954021 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:55:49.954096 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:49.954100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:49.954170 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:49.954299 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:49.954656 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.955446 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.956690 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.957210 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.957295 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:49.957722 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:49.958425 (+ 703us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:49.958528 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:49.958615 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 13:55:49.958671 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":563,"threads_started":1}
I20260927 13:55:49.973174 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:49.968111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:53477 (local address 127.22.26.193:40255)
0927 13:55:49.968333 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:49.968338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:49.968560 (+ 222us) server_negotiation.cc:407] Connection header received
0927 13:55:49.968699 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:49.968703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:49.968772 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:49.968862 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:49.969284 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.969997 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.971501 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.971891 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.971985 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:49.972418 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:49.972590 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:49.972666 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:49.972848 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 13:55:49.972932 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 13:55:49.973243 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:49.967827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40255 (local address 127.22.26.195:53477)
0927 13:55:49.968436 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:49.968456 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:49.968576 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:49.968903 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:49.968906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:49.968916 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:55:49.969139 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:49.969146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.970114 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.970117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:49.971327 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:49.971341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:49.972050 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:49.972058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:49.972204 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:49.972252 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:55:49.972683 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:55:49.972692 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 13:55:49.972773 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 13:55:49.972932 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":279,"threads_started":1}
I20260927 13:55:49.981391 24690 raft_consensus.cc:1261] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Refusing update from remote peer df6ff73877284524af9d77757f996281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:49.981441 24550 raft_consensus.cc:1261] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Refusing update from remote peer df6ff73877284524af9d77757f996281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:49.982026 25001 consensus_queue.cc:1035] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:49.982388 24996 consensus_queue.cc:1035] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:50.008224 25004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:50.001978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:46849 (local address 127.22.26.195:45049)
0927 13:55:50.002215 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:50.002221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:50.002677 (+ 456us) server_negotiation.cc:407] Connection header received
0927 13:55:50.002870 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:50.002875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:50.002942 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:50.003030 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:50.003522 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:50.004346 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:50.006288 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:50.006948 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:50.007112 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:50.007531 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:50.007659 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:50.007752 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:50.007898 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 13:55:50.007975 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 13:55:50.008544 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:50.001929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:45049 (local address 127.22.26.193:46849)
0927 13:55:50.002587 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:50.002602 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:50.002743 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:50.003048 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:50.003052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:50.003063 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:55:50.003387 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:50.003396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:50.004505 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:50.004510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:50.006144 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:50.006157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:50.007167 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:50.007174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:50.007315 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:50.007358 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:55:50.007767 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:55:50.007779 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 13:55:50.007872 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 13:55:50.008012 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":356,"threads_started":1}
I20260927 13:55:50.063369 24388 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:55:50.063638 24388 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:55:50.067206 24388 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923: Sending DeleteTablet for 3 replicas of tablet 0f5693cd0ec846ec92b20e261f11d48d
I20260927 13:55:50.068322 24525 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f5693cd0ec846ec92b20e261f11d48d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:50 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:56360
I20260927 13:55:50.068324 24804 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f5693cd0ec846ec92b20e261f11d48d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:50 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:32940
I20260927 13:55:50.068661 24981 tablet_replica.cc:330] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281: stopping tablet replica
I20260927 13:55:50.068728 24665 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f5693cd0ec846ec92b20e261f11d48d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:50 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:52716
I20260927 13:55:50.068902 24981 raft_consensus.cc:2229] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:55:50.068949 24980 tablet_replica.cc:330] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034: stopping tablet replica
I20260927 13:55:50.068995 24979 tablet_replica.cc:330] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0: stopping tablet replica
I20260927 13:55:50.069137 24980 raft_consensus.cc:2229] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:55:50.069203 24979 raft_consensus.cc:2229] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:55:50.069360 24980 raft_consensus.cc:2258] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:50.069422 24979 raft_consensus.cc:2258] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:50.069445 24981 raft_consensus.cc:2258] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:50.070663 24979 ts_tablet_manager.cc:1905] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:50.070758 24981 ts_tablet_manager.cc:1905] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:50.071256 24388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:55:50.072063 24388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:55:50.074004 24980 ts_tablet_manager.cc:1905] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:50.079530 24979 ts_tablet_manager.cc:1918] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:55:50.079695 24979 log.cc:1196] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/0f5693cd0ec846ec92b20e261f11d48d
I20260927 13:55:50.080199 24979 ts_tablet_manager.cc:1939] T 0f5693cd0ec846ec92b20e261f11d48d P 65f7a8c213d54ff8a7768f887c1be0a0: Deleting consensus metadata
I20260927 13:55:50.080710 24665 tablet_service.cc:1467] Processing CreateTablet for tablet 896c64fec4484407a1d5624eca85fef8 (DEFAULT_TABLE table=test-table [id=2c30d7ee23244694a8b1325bf00e50a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:50.081169 24665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896c64fec4484407a1d5624eca85fef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:50.081911 24981 ts_tablet_manager.cc:1918] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:55:50.082078 24981 log.cc:1196] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/0f5693cd0ec846ec92b20e261f11d48d
I20260927 13:55:50.082557 24981 ts_tablet_manager.cc:1939] T 0f5693cd0ec846ec92b20e261f11d48d P df6ff73877284524af9d77757f996281: Deleting consensus metadata
I20260927 13:55:50.087370 24980 ts_tablet_manager.cc:1918] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:55:50.087566 24980 log.cc:1196] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/0f5693cd0ec846ec92b20e261f11d48d
I20260927 13:55:50.087491 24525 tablet_service.cc:1467] Processing CreateTablet for tablet 896c64fec4484407a1d5624eca85fef8 (DEFAULT_TABLE table=test-table [id=2c30d7ee23244694a8b1325bf00e50a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:50.087507 24373 catalog_manager.cc:4841] TS 65f7a8c213d54ff8a7768f887c1be0a0 (127.22.26.194:44001): tablet 0f5693cd0ec846ec92b20e261f11d48d (table test-table [id=69157363d9164abf87e5c6fc68765e65]) successfully deleted
I20260927 13:55:50.087955 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896c64fec4484407a1d5624eca85fef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:50.088032 24980 ts_tablet_manager.cc:1939] T 0f5693cd0ec846ec92b20e261f11d48d P 5b22a74807ab439ea1954c5528293034: Deleting consensus metadata
I20260927 13:55:50.088105 24376 catalog_manager.cc:4841] TS df6ff73877284524af9d77757f996281 (127.22.26.195:45049): tablet 0f5693cd0ec846ec92b20e261f11d48d (table test-table [id=69157363d9164abf87e5c6fc68765e65]) successfully deleted
I20260927 13:55:50.089951 24375 catalog_manager.cc:4841] TS 5b22a74807ab439ea1954c5528293034 (127.22.26.193:40255): tablet 0f5693cd0ec846ec92b20e261f11d48d (table test-table [id=69157363d9164abf87e5c6fc68765e65]) successfully deleted
I20260927 13:55:50.090550 24804 tablet_service.cc:1467] Processing CreateTablet for tablet 896c64fec4484407a1d5624eca85fef8 (DEFAULT_TABLE table=test-table [id=2c30d7ee23244694a8b1325bf00e50a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:55:50.091018 24804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896c64fec4484407a1d5624eca85fef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:50.093516 24991 tablet_bootstrap.cc:492] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0: Bootstrap starting.
I20260927 13:55:50.094195 24992 tablet_bootstrap.cc:492] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034: Bootstrap starting.
I20260927 13:55:50.095631 24991 tablet_bootstrap.cc:654] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:50.095923 24992 tablet_bootstrap.cc:654] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:50.106448 24990 tablet_bootstrap.cc:492] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281: Bootstrap starting.
I20260927 13:55:50.108296 24990 tablet_bootstrap.cc:654] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:50.108306 24991 tablet_bootstrap.cc:492] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0: No bootstrap required, opened a new log
I20260927 13:55:50.108497 24991 ts_tablet_manager.cc:1397] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.012s
I20260927 13:55:50.109171 24992 tablet_bootstrap.cc:492] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034: No bootstrap required, opened a new log
I20260927 13:55:50.109500 24992 ts_tablet_manager.cc:1397] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.003s
I20260927 13:55:50.109344 24991 raft_consensus.cc:348] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:50.109620 24991 raft_consensus.cc:374] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:50.109725 24991 raft_consensus.cc:729] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f7a8c213d54ff8a7768f887c1be0a0, State: Initialized, Role: FOLLOWER
I20260927 13:55:50.110014 24991 consensus_queue.cc:260] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:50.110818 24992 raft_consensus.cc:348] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:50.111276 24992 raft_consensus.cc:374] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:50.111423 24992 raft_consensus.cc:729] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b22a74807ab439ea1954c5528293034, State: Initialized, Role: FOLLOWER
I20260927 13:55:50.111876 24992 consensus_queue.cc:260] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:50.113190 24991 ts_tablet_manager.cc:1428] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260927 13:55:50.113657 24992 ts_tablet_manager.cc:1428] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 13:55:50.116076 24990 tablet_bootstrap.cc:492] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281: No bootstrap required, opened a new log
I20260927 13:55:50.116478 24990 ts_tablet_manager.cc:1397] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260927 13:55:50.117125 24990 raft_consensus.cc:348] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:50.117363 24990 raft_consensus.cc:374] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:50.117477 24990 raft_consensus.cc:729] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6ff73877284524af9d77757f996281, State: Initialized, Role: FOLLOWER
I20260927 13:55:50.117729 24990 consensus_queue.cc:260] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:50.118261 24990 ts_tablet_manager.cc:1428] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 13:55:50.272073 24957 raft_consensus.cc:482] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:55:50.272348 24957 raft_consensus.cc:504] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:50.273269 24957 leader_election.cc:288] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b22a74807ab439ea1954c5528293034 (127.22.26.193:40255), df6ff73877284524af9d77757f996281 (127.22.26.195:45049)
I20260927 13:55:50.273917 24550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896c64fec4484407a1d5624eca85fef8" candidate_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b22a74807ab439ea1954c5528293034" is_pre_election: true
I20260927 13:55:50.273955 24829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896c64fec4484407a1d5624eca85fef8" candidate_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6ff73877284524af9d77757f996281" is_pre_election: true
I20260927 13:55:50.274202 24550 raft_consensus.cc:2454] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65f7a8c213d54ff8a7768f887c1be0a0 in term 0.
I20260927 13:55:50.274237 24829 raft_consensus.cc:2454] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65f7a8c213d54ff8a7768f887c1be0a0 in term 0.
I20260927 13:55:50.274753 24619 leader_election.cc:302] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65f7a8c213d54ff8a7768f887c1be0a0, df6ff73877284524af9d77757f996281; no voters:
I20260927 13:55:50.275070 24957 raft_consensus.cc:2781] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:55:50.275228 24957 raft_consensus.cc:482] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:55:50.275327 24957 raft_consensus.cc:3037] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:50.277527 24957 raft_consensus.cc:504] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:50.278191 24957 leader_election.cc:288] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [CANDIDATE]: Term 1 election: Requested vote from peers 5b22a74807ab439ea1954c5528293034 (127.22.26.193:40255), df6ff73877284524af9d77757f996281 (127.22.26.195:45049)
I20260927 13:55:50.278671 24550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896c64fec4484407a1d5624eca85fef8" candidate_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b22a74807ab439ea1954c5528293034"
I20260927 13:55:50.278713 24829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896c64fec4484407a1d5624eca85fef8" candidate_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6ff73877284524af9d77757f996281"
I20260927 13:55:50.278825 24550 raft_consensus.cc:3037] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:50.278908 24829 raft_consensus.cc:3037] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:50.280864 24550 raft_consensus.cc:2454] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65f7a8c213d54ff8a7768f887c1be0a0 in term 1.
I20260927 13:55:50.281256 24618 leader_election.cc:302] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b22a74807ab439ea1954c5528293034, 65f7a8c213d54ff8a7768f887c1be0a0; no voters:
I20260927 13:55:50.281517 24957 raft_consensus.cc:2781] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:50.281956 24957 raft_consensus.cc:686] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 65f7a8c213d54ff8a7768f887c1be0a0, State: Running, Role: LEADER
I20260927 13:55:50.282263 24957 consensus_queue.cc:237] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } }
I20260927 13:55:50.282480 24829 raft_consensus.cc:2454] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65f7a8c213d54ff8a7768f887c1be0a0 in term 1.
I20260927 13:55:50.285087 24388 catalog_manager.cc:5495] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65f7a8c213d54ff8a7768f887c1be0a0 (127.22.26.194). New cstate: current_term: 1 leader_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65f7a8c213d54ff8a7768f887c1be0a0" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 } health_report { overall_health: UNKNOWN } } }
I20260927 13:55:50.352898 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:50.347271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.22.26.194:44001)
0927 13:55:50.347491 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:50.347497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:50.347738 (+ 241us) server_negotiation.cc:407] Connection header received
0927 13:55:50.347921 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:50.347924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:50.347986 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:50.348060 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:50.348536 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:50.349539 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:50.350955 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:50.351603 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:50.351746 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:50.352123 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:50.352273 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:50.352387 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:50.352543 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 13:55:50.352628 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:55:50.362233 24550 raft_consensus.cc:1261] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Refusing update from remote peer 65f7a8c213d54ff8a7768f887c1be0a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:50.362233 24829 raft_consensus.cc:1261] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Refusing update from remote peer 65f7a8c213d54ff8a7768f887c1be0a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:50.362841 24957 consensus_queue.cc:1035] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b22a74807ab439ea1954c5528293034" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:50.363175 25017 consensus_queue.cc:1035] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6ff73877284524af9d77757f996281" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 13:55:50.399807 24962 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 13:55:51.987751 24388 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:55:51.987972 24388 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:55:51.991923 24388 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b223d7d3c86a4d8cab513a963f1e1923: Sending DeleteTablet for 3 replicas of tablet 896c64fec4484407a1d5624eca85fef8
I20260927 13:55:51.992714 24525 tablet_service.cc:1514] Processing DeleteTablet for tablet 896c64fec4484407a1d5624eca85fef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:51 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:56360
I20260927 13:55:51.992722 24665 tablet_service.cc:1514] Processing DeleteTablet for tablet 896c64fec4484407a1d5624eca85fef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:51 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:52716
I20260927 13:55:51.992985 24804 tablet_service.cc:1514] Processing DeleteTablet for tablet 896c64fec4484407a1d5624eca85fef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:55:51 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:32940
I20260927 13:55:51.993613 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 24460
I20260927 13:55:52.004026 25045 tablet_replica.cc:330] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034: stopping tablet replica
I20260927 13:55:52.004267 25045 raft_consensus.cc:2229] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:55:52.004339 25046 tablet_replica.cc:330] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281: stopping tablet replica
I20260927 13:55:52.004527 25045 raft_consensus.cc:2258] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:52.004648 25046 raft_consensus.cc:2229] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:55:52.004930 25046 raft_consensus.cc:2258] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:52.005499 25044 tablet_replica.cc:330] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0: stopping tablet replica
I20260927 13:55:52.005724 25044 raft_consensus.cc:2229] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:55:52.006126 25044 raft_consensus.cc:2258] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:55:52.006740 25046 ts_tablet_manager.cc:1905] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:52.007265 25044 ts_tablet_manager.cc:1905] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:52.007256 25045 ts_tablet_manager.cc:1905] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:55:52.013890 25044 ts_tablet_manager.cc:1918] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:55:52.014022 25044 log.cc:1196] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/896c64fec4484407a1d5624eca85fef8
I20260927 13:55:52.014252 25046 ts_tablet_manager.cc:1918] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:55:52.014428 25046 log.cc:1196] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/896c64fec4484407a1d5624eca85fef8
I20260927 13:55:52.014488 25044 ts_tablet_manager.cc:1939] T 896c64fec4484407a1d5624eca85fef8 P 65f7a8c213d54ff8a7768f887c1be0a0: Deleting consensus metadata
I20260927 13:55:52.014951 25046 ts_tablet_manager.cc:1939] T 896c64fec4484407a1d5624eca85fef8 P df6ff73877284524af9d77757f996281: Deleting consensus metadata
I20260927 13:55:52.015614 24373 catalog_manager.cc:4841] TS 65f7a8c213d54ff8a7768f887c1be0a0 (127.22.26.194:44001): tablet 896c64fec4484407a1d5624eca85fef8 (table test-table [id=2c30d7ee23244694a8b1325bf00e50a8]) successfully deleted
I20260927 13:55:52.016477 24376 catalog_manager.cc:4841] TS df6ff73877284524af9d77757f996281 (127.22.26.195:45049): tablet 896c64fec4484407a1d5624eca85fef8 (table test-table [id=2c30d7ee23244694a8b1325bf00e50a8]) successfully deleted
I20260927 13:55:52.019662 25045 ts_tablet_manager.cc:1918] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:55:52.019824 25045 log.cc:1196] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/896c64fec4484407a1d5624eca85fef8
I20260927 13:55:52.020406 25045 ts_tablet_manager.cc:1939] T 896c64fec4484407a1d5624eca85fef8 P 5b22a74807ab439ea1954c5528293034: Deleting consensus metadata
Sep 27 13:55:52 dist-test-slave-gkks krb5kdc[24344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:55:52.021559 24375 catalog_manager.cc:4841] TS 5b22a74807ab439ea1954c5528293034 (127.22.26.193:40255): tablet 896c64fec4484407a1d5624eca85fef8 (table test-table [id=2c30d7ee23244694a8b1325bf00e50a8]) successfully deleted
I20260927 13:55:52.028620 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:51.998777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.22.26.193:40255)
0927 13:55:52.003875 (+ 5098us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:52.003883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:52.003908 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:55:52.003969 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:52.003973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:52.004050 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:52.004274 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:52.007800 (+ 3526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:52.008890 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:52.010784 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:52.011503 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:52.011624 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:52.022085 (+ 10461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:52.022099 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:52.022115 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:52.022145 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:52.025600 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:52.026251 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:52.026260 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:52.026264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:52.026397 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:52.026871 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:52.026876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:52.026879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:52.027329 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:52.027630 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:52.028006 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 13:55:52.028215 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4953,"thread_start_us":4783,"threads_started":1}
W20260927 13:55:52.029321 24591 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36218
W20260927 13:55:52.029898 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 24460: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:55:52.030118 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 24460
I20260927 13:55:52.066272 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 24600
Sep 27 13:55:52 dist-test-slave-gkks krb5kdc[24344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:55:52.081328 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:52.066987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.22.26.194:44001)
0927 13:55:52.067514 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:52.067519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:52.067535 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:55:52.067595 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:52.067599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:52.067659 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:52.067776 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:52.069024 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:52.069808 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:52.071016 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:52.071549 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:52.071680 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:52.075679 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:52.075691 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:52.075706 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:52.075758 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:52.078638 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:52.079347 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:52.079354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:52.079357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:52.079479 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:52.079849 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:52.079853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:52.079855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:52.080176 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:52.080491 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:52.080762 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 13:55:52.081002 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":246,"threads_started":1}
W20260927 13:55:52.081946 24730 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56326
W20260927 13:55:52.082576 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 24600: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:55:52.082721 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 24600
I20260927 13:55:52.104252 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 24739
Sep 27 13:55:52 dist-test-slave-gkks krb5kdc[24344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:55:52.118623 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:52.105024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.22.26.195:45049)
0927 13:55:52.105496 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:52.105501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:52.105518 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:55:52.105666 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:52.105670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:52.105735 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:52.105841 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:52.107179 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:52.107905 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:52.109085 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:52.109470 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:52.109563 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:52.113693 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:52.113706 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:52.113722 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:52.113755 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:52.116421 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:52.116997 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:52.117004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:52.117006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:52.117120 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:52.117408 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:52.117411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:52.117413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:52.117707 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:52.117934 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:52.118108 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 13:55:52.118264 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":231,"threads_started":1}
W20260927 13:55:52.119390 24869 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34448
W20260927 13:55:52.119913 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 24739: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:55:52.120088 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 24739
I20260927 13:55:52.141456 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 24357
I20260927 13:55:52.142565 24429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:52.264806 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 24357
2026-09-27T13:55:52Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8578 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:55:52 dist-test-slave-gkks krb5kdc[25054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:55:52 dist-test-slave-gkks krb5kdc[25054](info): set up 2 sockets
Sep 27 13:55:52 dist-test-slave-gkks krb5kdc[25054](info): commencing operation
krb5kdc: starting...
W20260927 13:55:53.351820 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:55:53 dist-test-slave-gkks krb5kdc[25054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:55:53.540798 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:38349
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:38349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:53.860570 25067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:53.860872 25067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:53.861073 25067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:53.869068 25067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:53.869199 25067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:53.869282 25067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:53.869385 25067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:53.869468 25067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:53.879345 25067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:38349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:38349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:53.880681 25067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:53.882010 25067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:53.894213 25072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:53.894363 25067 server_base.cc:1034] running on GCE node
W20260927 13:55:53.894412 25073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:53.894232 25075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:53.895547 25067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:53.896901 25067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:53.898111 25067 hybrid_clock.cc:648] HybridClock initialized: now 1790517353898050 us; error 45 us; skew 500 ppm
I20260927 13:55:53.898424 25067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:53 dist-test-slave-gkks krb5kdc[25054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517353, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:53.901546 25067 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:55:53.903251 25067 webserver.cc:460] Webserver started at http://127.22.26.254:43005/ using document root <none> and password file <none>
I20260927 13:55:53.903970 25067 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:53.904096 25067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:53.904425 25067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:53.907521 25067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "1cca33fc34c34e73a442274df7949227"
format_stamp: "Formatted at 2026-09-27 13:55:53 on dist-test-slave-gkks"
server_key: "098503842cbe9578094d39fc376eb2e9"
server_key_iv: "e03ef5990f682ddbbb25fb5c54404328"
server_key_version: "encryptionkey@0"
I20260927 13:55:53.908293 25067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "1cca33fc34c34e73a442274df7949227"
format_stamp: "Formatted at 2026-09-27 13:55:53 on dist-test-slave-gkks"
server_key: "098503842cbe9578094d39fc376eb2e9"
server_key_iv: "e03ef5990f682ddbbb25fb5c54404328"
server_key_version: "encryptionkey@0"
I20260927 13:55:53.913534 25067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:55:53.917172 25082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:53.918604 25067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:55:53.918820 25067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "1cca33fc34c34e73a442274df7949227"
format_stamp: "Formatted at 2026-09-27 13:55:53 on dist-test-slave-gkks"
server_key: "098503842cbe9578094d39fc376eb2e9"
server_key_iv: "e03ef5990f682ddbbb25fb5c54404328"
server_key_version: "encryptionkey@0"
I20260927 13:55:53.919045 25067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:53.933071 25067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:53.937379 25067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:53.937682 25067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:53.956264 25067 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:38349
I20260927 13:55:53.956271 25133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:38349 every 8 connection(s)
I20260927 13:55:53.957705 25067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:55:53.958899 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 25067
I20260927 13:55:53.959182 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:55:53.959990 22635 external_mini_cluster.cc:1428] Setting key 23af29ae0694bf52236713d61d4498c3
I20260927 13:55:53.962598 25134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:55:53 dist-test-slave-gkks krb5kdc[25054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:53.976089 25134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227: Bootstrap starting.
I20260927 13:55:53.979579 25134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:53.980589 25134 log.cc:824] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:53.983444 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:53.962148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.22.26.254:38349)
0927 13:55:53.962974 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:53.962986 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:53.963026 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:55:53.964727 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:53.964749 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:53.965186 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:53.965535 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:53.967343 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:53.968780 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:53.970544 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:53.971306 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:53.971436 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:53.976158 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:53.976166 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:53.976181 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:53.976243 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:53.979358 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:53.980106 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:53.980113 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:53.980119 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:53.980277 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:53.980665 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:53.980672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:53.980679 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:53.981220 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:53.981524 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:53.981970 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 13:55:53.982409 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":362,"threads_started":1}
I20260927 13:55:53.983583 25134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227: No bootstrap required, opened a new log
I20260927 13:55:53.991168 25134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cca33fc34c34e73a442274df7949227" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 38349 } }
I20260927 13:55:53.991583 25134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:53.991714 25134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cca33fc34c34e73a442274df7949227, State: Initialized, Role: FOLLOWER
I20260927 13:55:53.992268 25134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cca33fc34c34e73a442274df7949227" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 38349 } }
I20260927 13:55:53.992508 25134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:55:53.992650 25134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:55:53.992825 25134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:53.994894 25134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cca33fc34c34e73a442274df7949227" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 38349 } }
I20260927 13:55:53.995489 25134 leader_election.cc:302] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cca33fc34c34e73a442274df7949227; no voters:
I20260927 13:55:53.996093 25134 leader_election.cc:288] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:55:53.996243 25139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:53.996904 25139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [term 1 LEADER]: Becoming Leader. State: Replica: 1cca33fc34c34e73a442274df7949227, State: Running, Role: LEADER
I20260927 13:55:53.997385 25139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cca33fc34c34e73a442274df7949227" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 38349 } }
I20260927 13:55:53.998099 25134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:54.000257 25141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cca33fc34c34e73a442274df7949227. Latest consensus state: current_term: 1 leader_uuid: "1cca33fc34c34e73a442274df7949227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cca33fc34c34e73a442274df7949227" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 38349 } } }
I20260927 13:55:54.000154 25140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cca33fc34c34e73a442274df7949227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cca33fc34c34e73a442274df7949227" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 38349 } } }
I20260927 13:55:54.000640 25141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:54.000691 25140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:54.003489 25147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:55:54.008428 25147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:55:54.017608 25147 catalog_manager.cc:1347] Generated new cluster ID: 5935d0a5243e47908c0a7a3904fdbfbc
I20260927 13:55:54.017777 25147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:55:54.035374 25147 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:55:54.037194 25147 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:55:54.045511 25147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cca33fc34c34e73a442274df7949227: Generated new TSK 0
I20260927 13:55:54.046347 25147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:55:54.103945 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:38349
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:54.429689 25160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:54.429948 25160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:54.430038 25160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:54.437247 25160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:54.437346 25160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:54.437457 25160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:55:54.446727 25160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:38349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:54.447973 25160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:54.449240 25160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:54.462347 25168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:54.462383 25166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:54.462986 25160 server_base.cc:1034] running on GCE node
W20260927 13:55:54.463130 25165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:54.463768 25160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:54.464697 25160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:54.465930 25160 hybrid_clock.cc:648] HybridClock initialized: now 1790517354465887 us; error 53 us; skew 500 ppm
I20260927 13:55:54.466240 25160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:54 dist-test-slave-gkks krb5kdc[25054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517354, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:54.469282 25160 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:55:54.470671 25160 webserver.cc:460] Webserver started at http://127.22.26.193:40835/ using document root <none> and password file <none>
I20260927 13:55:54.471379 25160 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:54.471517 25160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:54.471825 25160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:54.474751 25160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "b714328dd6144d85a62daf6768043074"
format_stamp: "Formatted at 2026-09-27 13:55:54 on dist-test-slave-gkks"
server_key: "87f9af7bbc8a98b83bb2679c0bc76077"
server_key_iv: "2a68a8711eb234c6d00327b6aea88d2d"
server_key_version: "encryptionkey@0"
I20260927 13:55:54.475520 25160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "b714328dd6144d85a62daf6768043074"
format_stamp: "Formatted at 2026-09-27 13:55:54 on dist-test-slave-gkks"
server_key: "87f9af7bbc8a98b83bb2679c0bc76077"
server_key_iv: "2a68a8711eb234c6d00327b6aea88d2d"
server_key_version: "encryptionkey@0"
I20260927 13:55:54.480496 25160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 13:55:54.485332 25175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:54.486840 25160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:55:54.487064 25160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "b714328dd6144d85a62daf6768043074"
format_stamp: "Formatted at 2026-09-27 13:55:54 on dist-test-slave-gkks"
server_key: "87f9af7bbc8a98b83bb2679c0bc76077"
server_key_iv: "2a68a8711eb234c6d00327b6aea88d2d"
server_key_version: "encryptionkey@0"
I20260927 13:55:54.487376 25160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:54.509768 25160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:54.514964 25160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:54.515321 25160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:54.516363 25160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:55:54.517767 25160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:54.517890 25160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:54.518049 25160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:54.518126 25160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:54.555130 25160 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:42167
I20260927 13:55:54.555174 25287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:42167 every 8 connection(s)
I20260927 13:55:54.556457 25160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:55:54.563978 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 25160
I20260927 13:55:54.564217 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:55:54.564898 22635 external_mini_cluster.cc:1428] Setting key add3855196a0b29211984db621ed4a5d
Sep 27 13:55:54 dist-test-slave-gkks krb5kdc[25054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517354, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:54.578450 25290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:54.559122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:52457 (local address 127.22.26.254:38349)
0927 13:55:54.559554 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:54.559559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:54.560587 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 13:55:54.562424 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:54.562429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:54.562521 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:54.562639 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:54.564518 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:54.565621 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:54.567589 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:54.567994 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:54.568103 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:54.572340 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:54.572345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:54.572348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:54.572378 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:54.575512 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:54.576208 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:54.576219 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:54.576222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:54.576333 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:54.576779 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:54.576784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:54.576789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:54.577048 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:54.577203 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:54.577898 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 13:55:54.578102 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":222,"threads_started":1}
I20260927 13:55:54.579527 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:54.559723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:38349 (local address 127.22.26.193:52457)
0927 13:55:54.560439 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:54.560477 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:54.562151 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:54.562872 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:54.562880 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:54.563447 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:54.564284 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:54.564312 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:54.565819 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:54.565824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:54.567435 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:54.567452 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:54.568241 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:54.568252 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:54.568610 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:54.569345 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:54.569383 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:54.572104 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:54.575647 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:54.575656 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:54.575670 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:54.576075 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:54.576466 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:54.576471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:54.576475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:54.576661 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:54.577184 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:54.577191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:54.577519 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 13:55:54.577773 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 13:55:54.578263 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":271,"threads_started":1}
I20260927 13:55:54.581086 25288 heartbeater.cc:347] Connected to a master server at 127.22.26.254:38349
I20260927 13:55:54.581388 25288 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:54.582063 25288 heartbeater.cc:511] Master 127.22.26.254:38349 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
I20260927 13:55:54.584152 25099 ts_manager.cc:194] Registered new tserver with Master: b714328dd6144d85a62daf6768043074 (127.22.26.193:42167)
I20260927 13:55:54.585594 25099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:52457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
I20260927 13:55:55.588505 25288 heartbeater.cc:503] Master 127.22.26.254:38349 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
W20260927 13:55:55.896646 22635 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.22.26.194: real 1.329s user 0.003s sys 0.007s
I20260927 13:55:55.897403 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:38349
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:56.219357 25294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:56.219617 25294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:56.219791 25294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:56.227149 25294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:56.227272 25294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:56.227455 25294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:55:56.237120 25294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:38349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:56.238358 25294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:56.239705 25294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:56.253643 25302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:56.253711 25299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:56.253676 25300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:56.254410 25294 server_base.cc:1034] running on GCE node
I20260927 13:55:56.254997 25294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:56.255972 25294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:56.257185 25294 hybrid_clock.cc:648] HybridClock initialized: now 1790517356257131 us; error 52 us; skew 500 ppm
I20260927 13:55:56.257501 25294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:56 dist-test-slave-gkks krb5kdc[25054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517356, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:56.260560 25294 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:55:56.261969 25294 webserver.cc:460] Webserver started at http://127.22.26.194:38689/ using document root <none> and password file <none>
I20260927 13:55:56.262630 25294 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:56.262770 25294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:56.263139 25294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:56.265924 25294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "fab9bd86df8944239f4892bc18ff2aef"
format_stamp: "Formatted at 2026-09-27 13:55:56 on dist-test-slave-gkks"
server_key: "3f1bd11c0c72cd071cefbe045dfdca28"
server_key_iv: "381ead4176f8cb50dc608ff3d5bcea6e"
server_key_version: "encryptionkey@0"
I20260927 13:55:56.266687 25294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "fab9bd86df8944239f4892bc18ff2aef"
format_stamp: "Formatted at 2026-09-27 13:55:56 on dist-test-slave-gkks"
server_key: "3f1bd11c0c72cd071cefbe045dfdca28"
server_key_iv: "381ead4176f8cb50dc608ff3d5bcea6e"
server_key_version: "encryptionkey@0"
I20260927 13:55:56.271687 25294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 13:55:56.274971 25309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:56.276409 25294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 13:55:56.276583 25294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "fab9bd86df8944239f4892bc18ff2aef"
format_stamp: "Formatted at 2026-09-27 13:55:56 on dist-test-slave-gkks"
server_key: "3f1bd11c0c72cd071cefbe045dfdca28"
server_key_iv: "381ead4176f8cb50dc608ff3d5bcea6e"
server_key_version: "encryptionkey@0"
I20260927 13:55:56.276926 25294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:56.291517 25294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:56.295549 25294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:56.295832 25294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:56.297057 25294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:55:56.298408 25294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:56.298507 25294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:56.298662 25294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:56.298738 25294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:56.331938 25294 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:45887
I20260927 13:55:56.332019 25421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:45887 every 8 connection(s)
I20260927 13:55:56.333211 25294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:55:56.335434 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 25294
I20260927 13:55:56.335590 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:55:56.336193 22635 external_mini_cluster.cc:1428] Setting key 1531fb362658e72d36c5942e77d7e002
Sep 27 13:55:56 dist-test-slave-gkks krb5kdc[25054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517356, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:56.354909 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:56.335794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:50283 (local address 127.22.26.254:38349)
0927 13:55:56.336311 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:56.336318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:56.337437 (+ 1119us) server_negotiation.cc:407] Connection header received
0927 13:55:56.339286 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:56.339295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:56.339412 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:56.339537 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:56.340939 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:56.342184 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:56.343793 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:56.344205 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:56.344366 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:56.347764 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:56.347773 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:56.347781 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:56.347824 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:56.351573 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:56.352339 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:56.352347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:56.352353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:56.352516 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:56.352944 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:56.352952 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:56.352958 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:56.353338 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:56.353507 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:56.354190 (+ 683us) server_negotiation.cc:299] Negotiation successful
0927 13:55:56.354442 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":253,"threads_started":1}
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
I20260927 13:55:56.355536 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:56.336462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:38349 (local address 127.22.26.194:50283)
0927 13:55:56.337278 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:56.337321 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:56.338969 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:56.339727 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:56.339736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:56.340232 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:56.340775 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:56.340792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:56.342345 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:56.342349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:56.343649 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:56.343662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:56.344458 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:56.344463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:56.344664 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:56.345362 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:56.345384 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:56.347546 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:56.351757 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:56.351763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:56.351795 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:56.352199 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:56.352637 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:56.352640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:56.352643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:56.352818 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:56.353467 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:56.353472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:56.353776 (+ 304us) client_negotiation.cc:769] Sending connection context
0927 13:55:56.354030 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 13:55:56.354409 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":343,"threads_started":1}
I20260927 13:55:56.356937 25422 heartbeater.cc:347] Connected to a master server at 127.22.26.254:38349
I20260927 13:55:56.357240 25422 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:56.357939 25422 heartbeater.cc:511] Master 127.22.26.254:38349 requested a full tablet report, sending...
I20260927 13:55:56.359333 25099 ts_manager.cc:194] Registered new tserver with Master: fab9bd86df8944239f4892bc18ff2aef (127.22.26.194:45887)
I20260927 13:55:56.360077 25099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:50283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:55:56.400251 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:38349
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:56.722375 25428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:56.722628 25428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:56.722817 25428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:56.729939 25428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:56.730031 25428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:56.730235 25428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:55:56.739615 25428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:38349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:56.740782 25428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:56.742022 25428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:56.754705 25433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:56.754858 25434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:56.755059 25436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:56.755877 25428 server_base.cc:1034] running on GCE node
I20260927 13:55:56.756516 25428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:56.757400 25428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:56.758625 25428 hybrid_clock.cc:648] HybridClock initialized: now 1790517356758591 us; error 50 us; skew 500 ppm
I20260927 13:55:56.758939 25428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:56 dist-test-slave-gkks krb5kdc[25054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517356, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:56.761864 25428 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:55:56.763281 25428 webserver.cc:460] Webserver started at http://127.22.26.195:45587/ using document root <none> and password file <none>
I20260927 13:55:56.763928 25428 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:56.764066 25428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:56.764372 25428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:56.767163 25428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "81f4aaaffc684686bc92e66972f77c68"
format_stamp: "Formatted at 2026-09-27 13:55:56 on dist-test-slave-gkks"
server_key: "f453d948ed75a2dd5d0e547c201b1605"
server_key_iv: "2a5d65589cf775e0053dc57b3be04806"
server_key_version: "encryptionkey@0"
I20260927 13:55:56.767922 25428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "81f4aaaffc684686bc92e66972f77c68"
format_stamp: "Formatted at 2026-09-27 13:55:56 on dist-test-slave-gkks"
server_key: "f453d948ed75a2dd5d0e547c201b1605"
server_key_iv: "2a5d65589cf775e0053dc57b3be04806"
server_key_version: "encryptionkey@0"
I20260927 13:55:56.772756 25428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:55:56.775951 25443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:56.777164 25428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 13:55:56.777328 25428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "81f4aaaffc684686bc92e66972f77c68"
format_stamp: "Formatted at 2026-09-27 13:55:56 on dist-test-slave-gkks"
server_key: "f453d948ed75a2dd5d0e547c201b1605"
server_key_iv: "2a5d65589cf775e0053dc57b3be04806"
server_key_version: "encryptionkey@0"
I20260927 13:55:56.777509 25428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:56.801416 25428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:56.805385 25428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:56.805668 25428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:56.806574 25428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:55:56.807917 25428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:55:56.808018 25428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:56.808167 25428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:55:56.808241 25428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:56.841044 25428 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:41375
I20260927 13:55:56.841089 25555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:41375 every 8 connection(s)
I20260927 13:55:56.842381 25428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:55:56.847975 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 25428
I20260927 13:55:56.848182 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:55:56.848748 22635 external_mini_cluster.cc:1428] Setting key de79f362c75f88f777247e560a313c2f
Sep 27 13:55:56 dist-test-slave-gkks krb5kdc[25054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517356, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:56.863644 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:56.844948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:47081 (local address 127.22.26.254:38349)
0927 13:55:56.845148 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:56.845153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:56.846301 (+ 1148us) server_negotiation.cc:407] Connection header received
0927 13:55:56.848305 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:56.848311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:56.848428 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:56.848541 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:56.850517 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:56.851307 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:56.853427 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:56.854167 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:56.854344 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:56.858423 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:56.858433 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:56.858438 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:56.858474 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:56.860905 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:56.861538 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:56.861541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:56.861543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:56.861656 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:56.862012 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:56.862015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:56.862017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:56.862276 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:56.862392 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:56.863036 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 13:55:56.863317 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 13:55:56.864665 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:56.845423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:38349 (local address 127.22.26.195:47081)
0927 13:55:56.846114 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:56.846169 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:56.847926 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:56.848779 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:56.848787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:56.849482 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:56.850344 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:56.850363 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:56.851458 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:56.851463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:56.853272 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:56.853284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:56.854405 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:56.854415 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:56.854903 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:56.855667 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:56.855695 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:56.858173 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:56.861023 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:56.861030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:56.861042 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:56.861439 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:56.861753 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:56.861756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:56.861758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:56.861923 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:56.862382 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:56.862388 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:56.862716 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 13:55:56.862941 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 13:55:56.863389 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":276,"threads_started":1}
I20260927 13:55:56.866005 25556 heartbeater.cc:347] Connected to a master server at 127.22.26.254:38349
I20260927 13:55:56.866309 25556 heartbeater.cc:464] Registering TS with master...
I20260927 13:55:56.867182 25556 heartbeater.cc:511] Master 127.22.26.254:38349 requested a full tablet report, sending...
I20260927 13:55:56.868609 25099 ts_manager.cc:194] Registered new tserver with Master: 81f4aaaffc684686bc92e66972f77c68 (127.22.26.195:41375)
I20260927 13:55:56.869431 25099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:47081
I20260927 13:55:56.877400 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 13:55:56.908167 25424 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:56.905658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.22.26.254:38349)
0927 13:55:56.905832 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:56.905837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:56.906247 (+ 410us) server_negotiation.cc:407] Connection header received
0927 13:55:56.906402 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:56.906406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:56.906477 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:56.906578 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:56.907942 (+ 1364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46142: BlockingRecv error: recv got EOF from 127.0.0.1:46142 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:55:56.911512 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 25160
W20260927 13:55:56.914418 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 25160: Not authorized: Client connection negotiation failed: client connection to 127.22.26.193:42167: server requires authentication, but client does not have Kerberos credentials available
I20260927 13:55:56.914633 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 25160
I20260927 13:55:56.931149 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 25294
W20260927 13:55:56.933967 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 25294: Not authorized: Client connection negotiation failed: client connection to 127.22.26.194:45887: server requires authentication, but client does not have Kerberos credentials available
I20260927 13:55:56.934131 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 25294
I20260927 13:55:56.950011 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 25428
W20260927 13:55:56.953179 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 25428: Not authorized: Client connection negotiation failed: client connection to 127.22.26.195:41375: server requires authentication, but client does not have Kerberos credentials available
I20260927 13:55:56.953356 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 25428
I20260927 13:55:56.969373 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 25067
I20260927 13:55:56.970608 25129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:57.074082 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 25067
2026-09-27T13:55:57Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4806 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:55:57 dist-test-slave-gkks krb5kdc[25573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:55:57 dist-test-slave-gkks krb5kdc[25573](info): set up 2 sockets
Sep 27 13:55:57 dist-test-slave-gkks krb5kdc[25573](info): commencing operation
krb5kdc: starting...
W20260927 13:55:58.171824 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:55:58 dist-test-slave-gkks krb5kdc[25573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:55:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:55:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:55:58.369040 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39403
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:39403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:58.688508 25586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:58.688831 25586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:58.689000 25586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:58.696111 25586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:58.696241 25586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:58.696349 25586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:58.696451 25586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:58.696537 25586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:58.706090 25586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:39403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:58.707436 25586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:58.708758 25586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:58.720405 25591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:58.720470 25594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:58.720438 25592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:58.721232 25586 server_base.cc:1034] running on GCE node
I20260927 13:55:58.722030 25586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:58.723443 25586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:58.724668 25586 hybrid_clock.cc:648] HybridClock initialized: now 1790517358724632 us; error 47 us; skew 500 ppm
I20260927 13:55:58.725046 25586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:58 dist-test-slave-gkks krb5kdc[25573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517358, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:58.728137 25586 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:55:58.729478 25586 webserver.cc:460] Webserver started at http://127.22.26.254:34417/ using document root <none> and password file <none>
I20260927 13:55:58.730156 25586 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:58.730298 25586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:58.730593 25586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:58.733445 25586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "a6a8a20484a740ee8df6b698c1249f8d"
format_stamp: "Formatted at 2026-09-27 13:55:58 on dist-test-slave-gkks"
server_key: "153456824db2e4663eee4cd08ac584d9"
server_key_iv: "0ddd4480597446423a2f775eb248dac6"
server_key_version: "encryptionkey@0"
I20260927 13:55:58.734218 25586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "a6a8a20484a740ee8df6b698c1249f8d"
format_stamp: "Formatted at 2026-09-27 13:55:58 on dist-test-slave-gkks"
server_key: "153456824db2e4663eee4cd08ac584d9"
server_key_iv: "0ddd4480597446423a2f775eb248dac6"
server_key_version: "encryptionkey@0"
I20260927 13:55:58.739254 25586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 13:55:58.742592 25601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:58.743937 25586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:55:58.744112 25586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "a6a8a20484a740ee8df6b698c1249f8d"
format_stamp: "Formatted at 2026-09-27 13:55:58 on dist-test-slave-gkks"
server_key: "153456824db2e4663eee4cd08ac584d9"
server_key_iv: "0ddd4480597446423a2f775eb248dac6"
server_key_version: "encryptionkey@0"
I20260927 13:55:58.744315 25586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:58.757762 25586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:58.762132 25586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:58.762509 25586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:58.782092 25586 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:39403
I20260927 13:55:58.782142 25652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:39403 every 8 connection(s)
I20260927 13:55:58.783622 25586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:55:58.787181 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 25586
I20260927 13:55:58.787465 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:55:58.787467 25653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:58.788156 22635 external_mini_cluster.cc:1428] Setting key 3f1e7ca86798ce4c14c466faa0efaef3
I20260927 13:55:58.801488 25653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d: Bootstrap starting.
Sep 27 13:55:58 dist-test-slave-gkks krb5kdc[25573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:55:58.805166 25653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:58.806555 25653 log.cc:824] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:58.809677 25653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d: No bootstrap required, opened a new log
I20260927 13:55:58.812230 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:58.790036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36276 (local address 127.22.26.254:39403)
0927 13:55:58.790882 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:58.790897 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:58.790945 (+ 48us) server_negotiation.cc:407] Connection header received
0927 13:55:58.792575 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:58.792605 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:58.793112 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:58.793573 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:58.794955 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:58.796890 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:58.798152 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:58.799280 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:58.799462 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:58.803774 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:58.803783 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:58.803800 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:58.803842 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:58.807685 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:58.808365 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:58.808373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:58.808380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:58.808593 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:58.809031 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:58.809036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:58.809039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:58.809730 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:58.810008 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:58.810469 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 13:55:58.810881 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":359,"threads_started":1}
I20260927 13:55:58.816092 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:55:58.819361 25653 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a8a20484a740ee8df6b698c1249f8d" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39403 } }
I20260927 13:55:58.819818 25653 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:58.819933 25653 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a8a20484a740ee8df6b698c1249f8d, State: Initialized, Role: FOLLOWER
I20260927 13:55:58.820551 25653 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a8a20484a740ee8df6b698c1249f8d" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39403 } }
I20260927 13:55:58.820840 25653 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:55:58.820979 25653 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:55:58.821153 25653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:58.824563 25653 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a8a20484a740ee8df6b698c1249f8d" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39403 } }
I20260927 13:55:58.825117 25653 leader_election.cc:302] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6a8a20484a740ee8df6b698c1249f8d; no voters:
I20260927 13:55:58.825790 25653 leader_election.cc:288] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:55:58.825886 25659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:58.827229 25659 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [term 1 LEADER]: Becoming Leader. State: Replica: a6a8a20484a740ee8df6b698c1249f8d, State: Running, Role: LEADER
I20260927 13:55:58.827838 25659 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a8a20484a740ee8df6b698c1249f8d" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39403 } }
I20260927 13:55:58.828223 25653 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:58.830775 25661 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6a8a20484a740ee8df6b698c1249f8d. Latest consensus state: current_term: 1 leader_uuid: "a6a8a20484a740ee8df6b698c1249f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a8a20484a740ee8df6b698c1249f8d" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39403 } } }
I20260927 13:55:58.831239 25661 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:58.831926 25660 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6a8a20484a740ee8df6b698c1249f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a8a20484a740ee8df6b698c1249f8d" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39403 } } }
I20260927 13:55:58.832247 25660 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:58.834506 25665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:55:58.842854 25665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:55:58.879383 25665 catalog_manager.cc:1347] Generated new cluster ID: 6149fdab4792453ab3f8a757e15f5f50
I20260927 13:55:58.879551 25665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:55:58.928439 25665 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:55:58.930532 25665 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:55:58.942649 25665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6a8a20484a740ee8df6b698c1249f8d: Generated new TSK 0
I20260927 13:55:58.943876 25665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:55:58.945640 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:58.924940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36288 (local address 127.22.26.254:39403)
0927 13:55:58.925185 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:58.925192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:58.927212 (+ 2020us) server_negotiation.cc:407] Connection header received
0927 13:55:58.927525 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:58.927532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:58.927635 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:58.927722 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:58.929310 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:58.930384 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:58.932233 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:58.932825 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:58.932914 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:58.943695 (+ 10781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:58.943702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:58.943706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:55:58.944588 (+ 882us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:58.944658 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:55:58.944688 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:58.944779 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:58.945043 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 13:55:58.945216 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":82}
I20260927 13:55:58.954833 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 25586
I20260927 13:55:58.956120 25648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:59.048161 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 25586
2026-09-27T13:55:59Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1968 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:55:59 dist-test-slave-gkks krb5kdc[25688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:55:59 dist-test-slave-gkks krb5kdc[25688](info): set up 2 sockets
Sep 27 13:55:59 dist-test-slave-gkks krb5kdc[25688](info): commencing operation
krb5kdc: starting...
W20260927 13:56:00.131732 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:56:00 dist-test-slave-gkks krb5kdc[25688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:56:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:56:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:56:00.324807 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39551
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:39551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:00.638646 25701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:00.639006 25701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:00.639195 25701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:00.646364 25701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:00.646512 25701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:00.646620 25701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:00.646723 25701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:00.646808 25701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:56:00.656759 25701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:39551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:00.658116 25701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:00.659513 25701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:00.671293 25709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:00.671334 25707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:00.671370 25706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:00.672183 25701 server_base.cc:1034] running on GCE node
I20260927 13:56:00.673014 25701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:00.674379 25701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:00.675639 25701 hybrid_clock.cc:648] HybridClock initialized: now 1790517360675593 us; error 51 us; skew 500 ppm
I20260927 13:56:00.676007 25701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:00 dist-test-slave-gkks krb5kdc[25688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517360, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:00.679013 25701 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:56:00.680413 25701 webserver.cc:460] Webserver started at http://127.22.26.254:44297/ using document root <none> and password file <none>
I20260927 13:56:00.681087 25701 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:00.681232 25701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:00.681538 25701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:00.684448 25701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "aa2a726363db422f850ef1321fc476f9"
format_stamp: "Formatted at 2026-09-27 13:56:00 on dist-test-slave-gkks"
server_key: "be99913536aecb77be0b77eda687ef19"
server_key_iv: "45f5ab8b84738cf2febbf4afc4f4d193"
server_key_version: "encryptionkey@0"
I20260927 13:56:00.685194 25701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "aa2a726363db422f850ef1321fc476f9"
format_stamp: "Formatted at 2026-09-27 13:56:00 on dist-test-slave-gkks"
server_key: "be99913536aecb77be0b77eda687ef19"
server_key_iv: "45f5ab8b84738cf2febbf4afc4f4d193"
server_key_version: "encryptionkey@0"
I20260927 13:56:00.690178 25701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:56:00.693392 25716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:00.694694 25701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:56:00.694919 25701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "aa2a726363db422f850ef1321fc476f9"
format_stamp: "Formatted at 2026-09-27 13:56:00 on dist-test-slave-gkks"
server_key: "be99913536aecb77be0b77eda687ef19"
server_key_iv: "45f5ab8b84738cf2febbf4afc4f4d193"
server_key_version: "encryptionkey@0"
I20260927 13:56:00.695127 25701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:00.733033 25701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:00.736943 25701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:00.737242 25701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:00.755316 25701 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:39551
I20260927 13:56:00.755334 25767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:39551 every 8 connection(s)
I20260927 13:56:00.756685 25701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:56:00.760375 25768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:00.761475 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 25701
I20260927 13:56:00.761732 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:56:00.762553 22635 external_mini_cluster.cc:1428] Setting key 94b3bb1f1c84e15d94215dc78cadc533
I20260927 13:56:00.774621 25768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9: Bootstrap starting.
Sep 27 13:56:00 dist-test-slave-gkks krb5kdc[25688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:00.778100 25768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:00.779337 25768 log.cc:824] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:00.782073 25768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9: No bootstrap required, opened a new log
I20260927 13:56:00.784844 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:00.764680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.22.26.254:39551)
0927 13:56:00.765478 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:00.765492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:00.765531 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:56:00.767055 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:00.767075 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:00.767503 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:00.767827 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:00.769487 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:00.770692 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:00.772467 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:00.773103 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:00.773229 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:00.777756 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:00.777763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:00.777777 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:00.777815 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:00.780858 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:00.781517 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:00.781529 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:00.781534 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:00.781688 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:00.782046 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:00.782051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:00.782055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:00.782662 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:00.782993 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:00.783424 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 13:56:00.783853 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":378,"threads_started":1}
I20260927 13:56:00.790982 25768 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2a726363db422f850ef1321fc476f9" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39551 } }
I20260927 13:56:00.791441 25768 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:00.791560 25768 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa2a726363db422f850ef1321fc476f9, State: Initialized, Role: FOLLOWER
I20260927 13:56:00.792117 25768 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2a726363db422f850ef1321fc476f9" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39551 } }
I20260927 13:56:00.792339 25768 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:56:00.792487 25768 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:56:00.792654 25768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:00.794837 25768 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2a726363db422f850ef1321fc476f9" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39551 } }
I20260927 13:56:00.795398 25768 leader_election.cc:302] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa2a726363db422f850ef1321fc476f9; no voters:
I20260927 13:56:00.796136 25768 leader_election.cc:288] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:56:00.796265 25773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:00.796995 25773 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [term 1 LEADER]: Becoming Leader. State: Replica: aa2a726363db422f850ef1321fc476f9, State: Running, Role: LEADER
I20260927 13:56:00.797533 25773 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2a726363db422f850ef1321fc476f9" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39551 } }
I20260927 13:56:00.798424 25768 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:00.800411 25774 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa2a726363db422f850ef1321fc476f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2a726363db422f850ef1321fc476f9" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39551 } } }
I20260927 13:56:00.800783 25774 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:00.803364 25775 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa2a726363db422f850ef1321fc476f9. Latest consensus state: current_term: 1 leader_uuid: "aa2a726363db422f850ef1321fc476f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2a726363db422f850ef1321fc476f9" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39551 } } }
I20260927 13:56:00.803849 25775 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:00.804672 25779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:00.809410 25779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:00.817035 25779 catalog_manager.cc:1347] Generated new cluster ID: 9878641928e14af5a9db7cd9c57f34ea
I20260927 13:56:00.817142 25779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:00.837062 25779 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:56:00.838905 25779 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:00.847404 25779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa2a726363db422f850ef1321fc476f9: Generated new TSK 0
I20260927 13:56:00.848196 25779 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:56:00.903613 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39551
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:01.228113 25794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:01.228459 25794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:01.228639 25794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:01.235951 25794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:01.236136 25794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:01.236343 25794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:56:01.246346 25794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:01.247751 25794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:01.249111 25794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:01.262585 25799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:01.263250 25802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:01.264945 25800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:01.265026 25794 server_base.cc:1034] running on GCE node
I20260927 13:56:01.265741 25794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:01.266851 25794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:01.268069 25794 hybrid_clock.cc:648] HybridClock initialized: now 1790517361267996 us; error 94 us; skew 500 ppm
I20260927 13:56:01.268420 25794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:01 dist-test-slave-gkks krb5kdc[25688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517361, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:01.271633 25794 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:56:01.273041 25794 webserver.cc:460] Webserver started at http://127.22.26.193:40745/ using document root <none> and password file <none>
I20260927 13:56:01.273705 25794 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:01.273813 25794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:01.274063 25794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:01.277016 25794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "70303429161943cda44a94a2a3538ee1"
format_stamp: "Formatted at 2026-09-27 13:56:01 on dist-test-slave-gkks"
server_key: "90a53bdbd9d92856ac00939971aa1e1d"
server_key_iv: "89a847a42b8426b367ae87a49643518d"
server_key_version: "encryptionkey@0"
I20260927 13:56:01.277732 25794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "70303429161943cda44a94a2a3538ee1"
format_stamp: "Formatted at 2026-09-27 13:56:01 on dist-test-slave-gkks"
server_key: "90a53bdbd9d92856ac00939971aa1e1d"
server_key_iv: "89a847a42b8426b367ae87a49643518d"
server_key_version: "encryptionkey@0"
I20260927 13:56:01.282732 25794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:56:01.286448 25809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:01.288056 25794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:01.288215 25794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "70303429161943cda44a94a2a3538ee1"
format_stamp: "Formatted at 2026-09-27 13:56:01 on dist-test-slave-gkks"
server_key: "90a53bdbd9d92856ac00939971aa1e1d"
server_key_iv: "89a847a42b8426b367ae87a49643518d"
server_key_version: "encryptionkey@0"
I20260927 13:56:01.288403 25794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:01.306329 25794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:01.310789 25794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:01.311036 25794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:01.312042 25794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:01.313340 25794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:01.313410 25794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:01.313495 25794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:01.313539 25794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:01.347256 25794 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:45341
I20260927 13:56:01.347342 25921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:45341 every 8 connection(s)
I20260927 13:56:01.348671 25794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:56:01.356381 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 25794
I20260927 13:56:01.356635 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:56:01.357322 22635 external_mini_cluster.cc:1428] Setting key ba8f11f1f3f3027c862ab9b35b803437
Sep 27 13:56:01 dist-test-slave-gkks krb5kdc[25688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517361, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:01.370326 25924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:01.351363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:56479 (local address 127.22.26.254:39551)
0927 13:56:01.351806 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:01.351811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:01.352923 (+ 1112us) server_negotiation.cc:407] Connection header received
0927 13:56:01.354770 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:01.354775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:01.354873 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:01.354992 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:01.356872 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:01.357679 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:01.359717 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:01.360115 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:01.360219 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:01.364532 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:01.364537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:01.364541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:01.364573 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:01.367272 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:01.368013 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:01.368026 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:01.368029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:01.368141 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:01.368592 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:01.368597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:01.368602 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:01.368865 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:01.369020 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:01.369727 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 13:56:01.369947 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":232,"threads_started":1}
I20260927 13:56:01.371441 25925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:01.352034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39551 (local address 127.22.26.193:56479)
0927 13:56:01.352761 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:01.352799 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:01.354495 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:01.355258 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:01.355267 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:01.355817 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:01.356669 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:01.356712 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:01.357851 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:01.357856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:01.359568 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:01.359582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:01.360350 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:01.360360 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:01.360718 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:01.361508 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:01.361548 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:01.364292 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:01.367411 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:01.367420 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:01.367438 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:01.367881 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:01.368255 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:01.368260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:01.368263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:01.368469 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:01.368994 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:01.369002 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:01.369350 (+ 348us) client_negotiation.cc:769] Sending connection context
0927 13:56:01.369624 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 13:56:01.370165 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":277,"threads_started":1}
I20260927 13:56:01.373137 25922 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39551
I20260927 13:56:01.373517 25922 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:01.374511 25922 heartbeater.cc:511] Master 127.22.26.254:39551 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
I20260927 13:56:01.376914 25733 ts_manager.cc:194] Registered new tserver with Master: 70303429161943cda44a94a2a3538ee1 (127.22.26.193:45341)
I20260927 13:56:01.378613 25733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:56479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:56:01.410820 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39551
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:01.735502 25928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:01.735767 25928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:01.735932 25928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:01.742921 25928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:01.743016 25928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:01.743218 25928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:56:01.752760 25928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:01.754031 25928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:01.755483 25928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:01.768226 25934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:01.768196 25936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:01.769121 25933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:01.769165 25928 server_base.cc:1034] running on GCE node
I20260927 13:56:01.769884 25928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:01.770781 25928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:01.772009 25928 hybrid_clock.cc:648] HybridClock initialized: now 1790517361771973 us; error 47 us; skew 500 ppm
I20260927 13:56:01.772310 25928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:01 dist-test-slave-gkks krb5kdc[25688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517361, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:01.775305 25928 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:56:01.776780 25928 webserver.cc:460] Webserver started at http://127.22.26.194:43109/ using document root <none> and password file <none>
I20260927 13:56:01.777431 25928 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:01.777571 25928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:01.777880 25928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:01.780732 25928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "15ff5f7c01cc4d28889fa6506202958c"
format_stamp: "Formatted at 2026-09-27 13:56:01 on dist-test-slave-gkks"
server_key: "a26e582772e50b65329e44024ff42085"
server_key_iv: "d970a62b1d77dc6bf4590cfd50c1a404"
server_key_version: "encryptionkey@0"
I20260927 13:56:01.781461 25928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "15ff5f7c01cc4d28889fa6506202958c"
format_stamp: "Formatted at 2026-09-27 13:56:01 on dist-test-slave-gkks"
server_key: "a26e582772e50b65329e44024ff42085"
server_key_iv: "d970a62b1d77dc6bf4590cfd50c1a404"
server_key_version: "encryptionkey@0"
I20260927 13:56:01.786397 25928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 13:56:01.789744 25943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:01.791234 25928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:01.791409 25928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "15ff5f7c01cc4d28889fa6506202958c"
format_stamp: "Formatted at 2026-09-27 13:56:01 on dist-test-slave-gkks"
server_key: "a26e582772e50b65329e44024ff42085"
server_key_iv: "d970a62b1d77dc6bf4590cfd50c1a404"
server_key_version: "encryptionkey@0"
I20260927 13:56:01.791610 25928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:01.802613 25928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:01.807060 25928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:01.807413 25928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:01.808306 25928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:01.809544 25928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:01.809646 25928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:01.809790 25928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:01.809860 25928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:01.842737 25928 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:35025
I20260927 13:56:01.842772 26055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:35025 every 8 connection(s)
I20260927 13:56:01.844116 25928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:56:01.853736 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 25928
I20260927 13:56:01.853961 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:56:01.854463 22635 external_mini_cluster.cc:1428] Setting key 8844720d58cf214f18b46e2865de0aaf
Sep 27 13:56:01 dist-test-slave-gkks krb5kdc[25688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517361, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:01.864429 25924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:01.847016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:55653 (local address 127.22.26.254:39551)
0927 13:56:01.847248 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:01.847252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:01.848647 (+ 1395us) server_negotiation.cc:407] Connection header received
0927 13:56:01.850383 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:01.850387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:01.850452 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:01.850542 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:01.852094 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:01.852789 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:01.854419 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:01.855020 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:01.855226 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:01.858804 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:01.858809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:01.858812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:01.858843 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:01.861448 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:01.862143 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:01.862149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:01.862153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:01.862287 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:01.862767 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:01.862771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:01.862773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:01.863040 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:01.863228 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:01.864019 (+ 791us) server_negotiation.cc:299] Negotiation successful
0927 13:56:01.864187 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 13:56:01.865903 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:01.847770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39551 (local address 127.22.26.194:55653)
0927 13:56:01.848490 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:01.848528 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:01.850141 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:01.850765 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:01.850773 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:01.851337 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:01.851938 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:01.851956 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:01.852946 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:01.852950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:01.854278 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:01.854294 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:01.855339 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:01.855347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:01.855554 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:01.856276 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:01.856322 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:01.858576 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:01.861587 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:01.861596 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:01.861612 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:01.862020 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:01.862401 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:01.862408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:01.862415 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:01.862647 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:01.863216 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:01.863227 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:01.863638 (+ 411us) client_negotiation.cc:769] Sending connection context
0927 13:56:01.863938 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 13:56:01.864493 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":251,"threads_started":1}
I20260927 13:56:01.867413 26056 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39551
I20260927 13:56:01.867733 26056 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:01.868431 26056 heartbeater.cc:511] Master 127.22.26.254:39551 requested a full tablet report, sending...
I20260927 13:56:01.869903 25733 ts_manager.cc:194] Registered new tserver with Master: 15ff5f7c01cc4d28889fa6506202958c (127.22.26.194:35025)
I20260927 13:56:01.870697 25733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:55653
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:56:01.897920 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39551
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:02.223683 26061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:02.223958 26061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:02.224124 26061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:02.231416 26061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:02.231523 26061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:02.231734 26061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:56:02.241377 26061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:02.242877 26061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:02.244246 26061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:02.257439 26066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:02.257405 26067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:02.257718 26069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:02.258543 26061 server_base.cc:1034] running on GCE node
I20260927 13:56:02.259212 26061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:02.260142 26061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:02.261384 26061 hybrid_clock.cc:648] HybridClock initialized: now 1790517362261326 us; error 45 us; skew 500 ppm
I20260927 13:56:02.261719 26061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:02 dist-test-slave-gkks krb5kdc[25688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517362, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:02.270605 26061 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:56:02.272050 26061 webserver.cc:460] Webserver started at http://127.22.26.195:33947/ using document root <none> and password file <none>
I20260927 13:56:02.272689 26061 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:02.272799 26061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:02.273036 26061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:02.275995 26061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "999d14df6a024705a8ec2710bd30be68"
format_stamp: "Formatted at 2026-09-27 13:56:02 on dist-test-slave-gkks"
server_key: "2f0b05dd4b65f9d55d6c96fc01107de3"
server_key_iv: "3ec7d95d0929332ef709a6ba6f838383"
server_key_version: "encryptionkey@0"
I20260927 13:56:02.276697 26061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "999d14df6a024705a8ec2710bd30be68"
format_stamp: "Formatted at 2026-09-27 13:56:02 on dist-test-slave-gkks"
server_key: "2f0b05dd4b65f9d55d6c96fc01107de3"
server_key_iv: "3ec7d95d0929332ef709a6ba6f838383"
server_key_version: "encryptionkey@0"
I20260927 13:56:02.281592 26061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:56:02.285082 26076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:02.286449 26061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 13:56:02.286620 26061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "999d14df6a024705a8ec2710bd30be68"
format_stamp: "Formatted at 2026-09-27 13:56:02 on dist-test-slave-gkks"
server_key: "2f0b05dd4b65f9d55d6c96fc01107de3"
server_key_iv: "3ec7d95d0929332ef709a6ba6f838383"
server_key_version: "encryptionkey@0"
I20260927 13:56:02.286784 26061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:02.303156 26061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:02.307331 26061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:02.307615 26061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:02.308560 26061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:02.309881 26061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:02.309950 26061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:02.310096 26061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:02.310143 26061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:02.342015 26061 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:44395
I20260927 13:56:02.342058 26188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:44395 every 8 connection(s)
I20260927 13:56:02.343313 26061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:56:02.345247 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 26061
I20260927 13:56:02.345466 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:56:02.346079 22635 external_mini_cluster.cc:1428] Setting key 05212ff7614fd3ff7746bcd62b3a57c9
Sep 27 13:56:02 dist-test-slave-gkks krb5kdc[25688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517362, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:02.364557 25924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:02.346131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:51421 (local address 127.22.26.254:39551)
0927 13:56:02.347228 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:02.347234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:02.347825 (+ 591us) server_negotiation.cc:407] Connection header received
0927 13:56:02.350411 (+ 2586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:02.350415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:02.350531 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:02.350635 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:02.352596 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:02.353402 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:02.354882 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:02.355349 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:02.355444 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:02.358606 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:02.358611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:02.358614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:02.358644 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:02.361896 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:02.362569 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:02.362573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:02.362575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:02.362692 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:02.363078 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:02.363082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:02.363085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:02.363443 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:02.363568 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:02.364165 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 13:56:02.364319 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946}
I20260927 13:56:02.365504 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:02.346752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39551 (local address 127.22.26.195:51421)
0927 13:56:02.347645 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:02.347701 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:02.350091 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:02.350906 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:02.350918 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:02.351648 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:02.352424 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:02.352447 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:02.353545 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:02.353549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:02.354753 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:02.354765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:02.355532 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:02.355537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:02.355729 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:02.356334 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:02.356358 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:02.358425 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:02.362007 (+ 3582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:02.362014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:02.362025 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:02.362435 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:02.362785 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:02.362789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:02.362791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:02.362959 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:02.363561 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:02.363566 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:02.363848 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 13:56:02.364075 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 13:56:02.364441 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":460,"threads_started":1}
I20260927 13:56:02.366806 26189 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39551
I20260927 13:56:02.367144 26189 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:02.367875 26189 heartbeater.cc:511] Master 127.22.26.254:39551 requested a full tablet report, sending...
I20260927 13:56:02.369243 25733 ts_manager.cc:194] Registered new tserver with Master: 999d14df6a024705a8ec2710bd30be68 (127.22.26.195:44395)
I20260927 13:56:02.370050 25733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:51421
I20260927 13:56:02.373521 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:56:02.381531 25922 heartbeater.cc:503] Master 127.22.26.254:39551 was elected leader, sending a full tablet report...
Sep 27 13:56:02 dist-test-slave-gkks krb5kdc[25688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517362, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 13:56:02 dist-test-slave-gkks krb5kdc[25688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517362, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:02.422693 25924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:02.408786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.22.26.254:39551)
0927 13:56:02.408971 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:02.408976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:02.409503 (+ 527us) server_negotiation.cc:407] Connection header received
0927 13:56:02.409640 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:02.409644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:02.409717 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:02.409811 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:02.411290 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:02.412117 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:02.413448 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:02.413850 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:02.413948 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:02.417821 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:02.417826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:02.417829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:02.417861 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:02.420416 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:02.421044 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:02.421049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:02.421051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:02.421201 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:02.421552 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:02.421556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:02.421558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:02.421891 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:02.422080 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:02.422324 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 13:56:02.422477 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 13:56:02.423343 25733 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36802
I20260927 13:56:02.424891 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 25794
Sep 27 13:56:02 dist-test-slave-gkks krb5kdc[25688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517362, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:56:02.440379 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:02.426331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.22.26.193:45341)
0927 13:56:02.426920 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:02.426927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:02.426952 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:56:02.427165 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:02.427175 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:02.427583 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:02.427746 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:02.428817 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:02.429829 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:02.430926 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:02.431537 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:02.431635 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:02.435458 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:02.435468 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:02.435481 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:02.435511 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:02.438093 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:02.438668 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:02.438674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:02.438676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:02.438809 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:02.439149 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:02.439155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:02.439159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:02.439487 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:02.439728 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:02.439929 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 13:56:02.440108 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":268,"threads_started":1}
W20260927 13:56:02.441603 25917 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35182
W20260927 13:56:02.442718 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 25794: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:56:02.442873 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 25794
I20260927 13:56:02.458966 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 25928
Sep 27 13:56:02 dist-test-slave-gkks krb5kdc[25688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517362, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:56:02.475045 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:02.459996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.22.26.194:35025)
0927 13:56:02.460500 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:02.460507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:02.460530 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:56:02.460587 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:02.460593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:02.460805 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:02.460975 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:02.462183 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:02.463243 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:02.464581 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:02.465131 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:02.465280 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:02.469283 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:02.469299 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:02.469316 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:02.469357 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:02.472308 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:02.472957 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:02.472964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:02.472967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:02.473086 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:02.473462 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:02.473468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:02.473471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:02.473795 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:02.474121 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:02.474425 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 13:56:02.474708 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":255,"threads_started":1}
W20260927 13:56:02.476378 26051 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58140
W20260927 13:56:02.477360 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 25928: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:56:02.477502 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 25928
I20260927 13:56:02.493191 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 26061
Sep 27 13:56:02 dist-test-slave-gkks krb5kdc[25688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517362, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:56:02.509490 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:02.494101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43904 (local address 127.22.26.195:44395)
0927 13:56:02.494678 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:02.494685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:02.494713 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:56:02.494771 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:02.494777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:02.494979 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:02.495173 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:02.496398 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:02.497503 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:02.499028 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:02.499822 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:02.499947 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:02.504091 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:02.504102 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:02.504115 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:02.504145 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:02.506957 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:02.507652 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:02.507659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:02.507661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:02.507797 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:02.508105 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:02.508108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:02.508110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:02.508426 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:02.508647 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:02.508928 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 13:56:02.509109 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":284,"threads_started":1}
W20260927 13:56:02.510715 26184 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43904
W20260927 13:56:02.511878 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 26061: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:56:02.512036 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 26061
I20260927 13:56:02.527168 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 25701
I20260927 13:56:02.528420 25763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:02.631392 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 25701
2026-09-27T13:56:02Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3588 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:56:02 dist-test-slave-gkks krb5kdc[26206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:56:02 dist-test-slave-gkks krb5kdc[26206](info): set up 2 sockets
Sep 27 13:56:02 dist-test-slave-gkks krb5kdc[26206](info): commencing operation
krb5kdc: starting...
W20260927 13:56:03.715824 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:56:03 dist-test-slave-gkks krb5kdc[26206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:56:03.910434 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:41217
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:41217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:04.225562 26219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:04.225883 26219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:04.226037 26219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:04.233063 26219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:04.233204 26219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:04.233323 26219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:04.233418 26219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:04.233505 26219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:56:04.242936 26219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:41217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:41217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:04.244288 26219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:04.245585 26219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:04.257302 26227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:04.257298 26225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:04.257877 26224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:04.258503 26219 server_base.cc:1034] running on GCE node
I20260927 13:56:04.259312 26219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:04.260711 26219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:04.261924 26219 hybrid_clock.cc:648] HybridClock initialized: now 1790517364261879 us; error 31 us; skew 500 ppm
I20260927 13:56:04.262270 26219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:04 dist-test-slave-gkks krb5kdc[26206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517364, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:04.265305 26219 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:56:04.266788 26219 webserver.cc:460] Webserver started at http://127.22.26.254:39479/ using document root <none> and password file <none>
I20260927 13:56:04.267531 26219 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:04.267661 26219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:04.267987 26219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:04.270879 26219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "525d849f29e04c71a5a7fa44f1900afd"
format_stamp: "Formatted at 2026-09-27 13:56:04 on dist-test-slave-gkks"
server_key: "9f77094c0c74ec58b1f3581158751add"
server_key_iv: "d6a5f51c6d84becf867c3fa35f92faea"
server_key_version: "encryptionkey@0"
I20260927 13:56:04.271718 26219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "525d849f29e04c71a5a7fa44f1900afd"
format_stamp: "Formatted at 2026-09-27 13:56:04 on dist-test-slave-gkks"
server_key: "9f77094c0c74ec58b1f3581158751add"
server_key_iv: "d6a5f51c6d84becf867c3fa35f92faea"
server_key_version: "encryptionkey@0"
I20260927 13:56:04.277784 26219 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 13:56:04.281112 26234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:04.282369 26219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:56:04.282533 26219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "525d849f29e04c71a5a7fa44f1900afd"
format_stamp: "Formatted at 2026-09-27 13:56:04 on dist-test-slave-gkks"
server_key: "9f77094c0c74ec58b1f3581158751add"
server_key_iv: "d6a5f51c6d84becf867c3fa35f92faea"
server_key_version: "encryptionkey@0"
I20260927 13:56:04.282702 26219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:04.303156 26219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:04.307284 26219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:04.307574 26219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:04.326319 26219 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:41217
I20260927 13:56:04.326339 26285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:41217 every 8 connection(s)
I20260927 13:56:04.327728 26219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:56:04.328557 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 26219
I20260927 13:56:04.328841 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:56:04.329617 22635 external_mini_cluster.cc:1428] Setting key b55d2366265ec6729bd9723b725f30f7
I20260927 13:56:04.332188 26286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:04.342509 26286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd: Bootstrap starting.
Sep 27 13:56:04 dist-test-slave-gkks krb5kdc[26206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:04.345968 26286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:04.347013 26286 log.cc:824] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:04.349792 26286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd: No bootstrap required, opened a new log
I20260927 13:56:04.351250 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:04.331619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.22.26.254:41217)
0927 13:56:04.332421 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:04.332434 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:04.332478 (+ 44us) server_negotiation.cc:407] Connection header received
0927 13:56:04.334157 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:04.334179 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:04.334560 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:04.334969 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:04.336448 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:04.337751 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:04.338968 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:04.339745 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:04.339962 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:04.343949 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:04.343959 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:04.343972 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:04.344014 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:04.347162 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:04.347976 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:04.347982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:04.347988 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:04.348146 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:04.348594 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:04.348599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:04.348601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:04.349095 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:04.349401 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:04.349849 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 13:56:04.350250 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":320,"threads_started":1}
I20260927 13:56:04.358217 26286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525d849f29e04c71a5a7fa44f1900afd" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 41217 } }
I20260927 13:56:04.358604 26286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:04.358713 26286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525d849f29e04c71a5a7fa44f1900afd, State: Initialized, Role: FOLLOWER
I20260927 13:56:04.359318 26286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525d849f29e04c71a5a7fa44f1900afd" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 41217 } }
I20260927 13:56:04.359551 26286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:56:04.359691 26286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:56:04.359879 26286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:04.362303 26286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525d849f29e04c71a5a7fa44f1900afd" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 41217 } }
I20260927 13:56:04.362787 26286 leader_election.cc:302] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 525d849f29e04c71a5a7fa44f1900afd; no voters:
I20260927 13:56:04.363420 26286 leader_election.cc:288] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:56:04.363552 26291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:04.364250 26291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [term 1 LEADER]: Becoming Leader. State: Replica: 525d849f29e04c71a5a7fa44f1900afd, State: Running, Role: LEADER
I20260927 13:56:04.364897 26291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525d849f29e04c71a5a7fa44f1900afd" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 41217 } }
I20260927 13:56:04.365813 26286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:04.367745 26292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "525d849f29e04c71a5a7fa44f1900afd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525d849f29e04c71a5a7fa44f1900afd" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 41217 } } }
I20260927 13:56:04.368204 26292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:04.368093 26293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 525d849f29e04c71a5a7fa44f1900afd. Latest consensus state: current_term: 1 leader_uuid: "525d849f29e04c71a5a7fa44f1900afd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525d849f29e04c71a5a7fa44f1900afd" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 41217 } } }
I20260927 13:56:04.368408 26293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:04.369849 26299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:04.375264 26299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:04.384068 26299 catalog_manager.cc:1347] Generated new cluster ID: dc94ac8554ce4422ad3ab5164fc76df9
I20260927 13:56:04.384188 26299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:04.402810 26299 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:56:04.404188 26299 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:04.416937 26299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 525d849f29e04c71a5a7fa44f1900afd: Generated new TSK 0
I20260927 13:56:04.417768 26299 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:56:04.464613 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:41217
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:04.783056 26312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:04.783448 26312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:04.783629 26312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:04.790874 26312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:04.790969 26312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:04.791195 26312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:56:04.800645 26312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:41217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:04.801898 26312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:04.803233 26312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:04.816085 26320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:04.816164 26317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:04.816200 26318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:04.817243 26312 server_base.cc:1034] running on GCE node
I20260927 13:56:04.817837 26312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:04.818856 26312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:04.820075 26312 hybrid_clock.cc:648] HybridClock initialized: now 1790517364820008 us; error 66 us; skew 500 ppm
I20260927 13:56:04.820384 26312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:04 dist-test-slave-gkks krb5kdc[26206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517364, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:04.823477 26312 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:56:04.824767 26312 webserver.cc:460] Webserver started at http://127.22.26.193:45863/ using document root <none> and password file <none>
I20260927 13:56:04.825357 26312 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:04.825448 26312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:04.825690 26312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:04.828420 26312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "4ff9bd6cd762431bb555c678ab2eb249"
format_stamp: "Formatted at 2026-09-27 13:56:04 on dist-test-slave-gkks"
server_key: "51cb74b83965c4532d038c277b2ff8ea"
server_key_iv: "046d1ab06dc1fdf8735222eb6458f552"
server_key_version: "encryptionkey@0"
I20260927 13:56:04.829126 26312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "4ff9bd6cd762431bb555c678ab2eb249"
format_stamp: "Formatted at 2026-09-27 13:56:04 on dist-test-slave-gkks"
server_key: "51cb74b83965c4532d038c277b2ff8ea"
server_key_iv: "046d1ab06dc1fdf8735222eb6458f552"
server_key_version: "encryptionkey@0"
I20260927 13:56:04.834478 26312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 13:56:04.837985 26327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:04.839319 26312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:04.839474 26312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "4ff9bd6cd762431bb555c678ab2eb249"
format_stamp: "Formatted at 2026-09-27 13:56:04 on dist-test-slave-gkks"
server_key: "51cb74b83965c4532d038c277b2ff8ea"
server_key_iv: "046d1ab06dc1fdf8735222eb6458f552"
server_key_version: "encryptionkey@0"
I20260927 13:56:04.839673 26312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:04.854177 26312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:04.858294 26312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:04.858577 26312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:04.859536 26312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:04.860898 26312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:04.860968 26312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:04.861131 26312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:04.861196 26312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:04.894449 26312 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:44993
I20260927 13:56:04.894665 26439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:44993 every 8 connection(s)
I20260927 13:56:04.895855 26312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:56:04.904120 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 26312
I20260927 13:56:04.904335 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:56:04.905105 22635 external_mini_cluster.cc:1428] Setting key 7be15e92134fee790729a60d5105d2c0
Sep 27 13:56:04 dist-test-slave-gkks krb5kdc[26206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517364, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:04.916932 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:04.898594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:37195 (local address 127.22.26.254:41217)
0927 13:56:04.899013 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:04.899017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:04.900148 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 13:56:04.901886 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:04.901890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:04.901959 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:04.902064 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:04.903571 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:04.904739 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:04.906462 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:04.907015 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:04.907171 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:04.910520 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:04.910529 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:04.910538 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:04.910583 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:04.913585 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:04.914489 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:04.914493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:04.914495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:04.914609 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:04.915063 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:04.915066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:04.915068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:04.915381 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:04.915492 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:04.916411 (+ 919us) server_negotiation.cc:299] Negotiation successful
0927 13:56:04.916657 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":208,"threads_started":1}
I20260927 13:56:04.918349 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:04.899262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:41217 (local address 127.22.26.193:37195)
0927 13:56:04.899981 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:04.900024 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:04.901619 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:04.902277 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:04.902284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:04.902762 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:04.903392 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:04.903410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:04.904904 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:04.904908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:04.906329 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:04.906341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:04.907263 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:04.907268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:04.907462 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:04.908125 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:04.908154 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:04.910300 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:04.913723 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:04.913734 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:04.913754 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:04.914382 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:04.914714 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:04.914721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:04.914728 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:04.914959 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:04.915508 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:04.915519 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:04.915987 (+ 468us) client_negotiation.cc:769] Sending connection context
0927 13:56:04.916300 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 13:56:04.916856 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":277,"threads_started":1}
I20260927 13:56:04.920037 26440 heartbeater.cc:347] Connected to a master server at 127.22.26.254:41217
I20260927 13:56:04.920333 26440 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:04.921125 26440 heartbeater.cc:511] Master 127.22.26.254:41217 requested a full tablet report, sending...
I20260927 13:56:04.923286 26251 ts_manager.cc:194] Registered new tserver with Master: 4ff9bd6cd762431bb555c678ab2eb249 (127.22.26.193:44993)
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
I20260927 13:56:04.924769 26251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:37195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:56:04.949996 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:41217
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:05.263053 26446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:05.263442 26446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:05.263605 26446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:05.270872 26446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:05.271016 26446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:05.271226 26446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:56:05.282343 26446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:41217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:05.283865 26446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:05.285181 26446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:05.298030 26451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:05.298030 26454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:05.298071 26452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:05.298888 26446 server_base.cc:1034] running on GCE node
I20260927 13:56:05.299543 26446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:05.300446 26446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:05.301674 26446 hybrid_clock.cc:648] HybridClock initialized: now 1790517365301643 us; error 58 us; skew 500 ppm
I20260927 13:56:05.301990 26446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:05 dist-test-slave-gkks krb5kdc[26206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517365, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:05.305104 26446 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:56:05.306556 26446 webserver.cc:460] Webserver started at http://127.22.26.194:45897/ using document root <none> and password file <none>
I20260927 13:56:05.307307 26446 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:05.307448 26446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:05.307760 26446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:05.310739 26446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "53ed3f0aecd6479b98475730681a8087"
format_stamp: "Formatted at 2026-09-27 13:56:05 on dist-test-slave-gkks"
server_key: "2c7fc9355bb679f459119e675f535e72"
server_key_iv: "b74dd0c64d45b1d4ed2751dcadd5eadc"
server_key_version: "encryptionkey@0"
I20260927 13:56:05.311645 26446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "53ed3f0aecd6479b98475730681a8087"
format_stamp: "Formatted at 2026-09-27 13:56:05 on dist-test-slave-gkks"
server_key: "2c7fc9355bb679f459119e675f535e72"
server_key_iv: "b74dd0c64d45b1d4ed2751dcadd5eadc"
server_key_version: "encryptionkey@0"
I20260927 13:56:05.316660 26446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:56:05.320009 26461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:05.321285 26446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:56:05.321460 26446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "53ed3f0aecd6479b98475730681a8087"
format_stamp: "Formatted at 2026-09-27 13:56:05 on dist-test-slave-gkks"
server_key: "2c7fc9355bb679f459119e675f535e72"
server_key_iv: "b74dd0c64d45b1d4ed2751dcadd5eadc"
server_key_version: "encryptionkey@0"
I20260927 13:56:05.321664 26446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:05.336012 26446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:05.340152 26446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:05.340458 26446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:05.341337 26446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:05.342556 26446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:05.342658 26446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:05.342803 26446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:05.342901 26446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:05.375952 26446 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:36393
I20260927 13:56:05.376000 26573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:36393 every 8 connection(s)
I20260927 13:56:05.377242 26446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:56:05.387339 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 26446
I20260927 13:56:05.387522 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:56:05.388094 22635 external_mini_cluster.cc:1428] Setting key 0655e31f719c53de733bb44d75797458
Sep 27 13:56:05 dist-test-slave-gkks krb5kdc[26206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517365, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:05.398051 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:05.379675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:33063 (local address 127.22.26.254:41217)
0927 13:56:05.379849 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:05.379854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:05.381009 (+ 1155us) server_negotiation.cc:407] Connection header received
0927 13:56:05.382788 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:05.382793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:05.382868 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:05.382958 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:05.384391 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.385101 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.386660 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.387191 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.387287 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:05.391111 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:05.391121 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:05.391129 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:05.391173 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:05.394842 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:05.395593 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:05.395601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:05.395608 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:05.395780 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:05.396183 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:05.396190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:05.396193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:05.396464 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:05.396582 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:05.397658 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0927 13:56:05.397824 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:56:05.399861 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:05.380134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:41217 (local address 127.22.26.194:33063)
0927 13:56:05.380825 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:05.380864 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:05.382518 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:05.383207 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:05.383215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:05.383702 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:05.384232 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:05.384248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.385216 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.385220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:05.386523 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:05.386535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.387417 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.387425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:05.387722 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:05.388494 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:05.388515 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:05.390861 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:05.394987 (+ 4126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:05.394994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:05.395006 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:05.395476 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:05.395889 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:05.395892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:05.395895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:05.396069 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:05.396590 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:05.396600 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:05.397095 (+ 495us) client_negotiation.cc:769] Sending connection context
0927 13:56:05.397555 (+ 460us) client_negotiation.cc:240] Negotiation successful
0927 13:56:05.398221 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":281,"threads_started":1}
I20260927 13:56:05.401407 26574 heartbeater.cc:347] Connected to a master server at 127.22.26.254:41217
I20260927 13:56:05.401721 26574 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:05.402438 26574 heartbeater.cc:511] Master 127.22.26.254:41217 requested a full tablet report, sending...
I20260927 13:56:05.403970 26251 ts_manager.cc:194] Registered new tserver with Master: 53ed3f0aecd6479b98475730681a8087 (127.22.26.194:36393)
I20260927 13:56:05.404770 26251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:33063
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:56:05.436443 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:41217
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:05.749565 26579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:05.749883 26579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:05.750051 26579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:05.757265 26579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:05.757427 26579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:05.757594 26579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:56:05.767478 26579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:41217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:05.768786 26579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:05.770090 26579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:05.783362 26584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:05.783332 26587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:05.783288 26585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:05.784034 26579 server_base.cc:1034] running on GCE node
I20260927 13:56:05.784610 26579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:05.785645 26579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:05.786867 26579 hybrid_clock.cc:648] HybridClock initialized: now 1790517365786824 us; error 33 us; skew 500 ppm
I20260927 13:56:05.787253 26579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:05 dist-test-slave-gkks krb5kdc[26206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517365, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:05.790207 26579 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:56:05.791644 26579 webserver.cc:460] Webserver started at http://127.22.26.195:43079/ using document root <none> and password file <none>
I20260927 13:56:05.792317 26579 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:05.792459 26579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:05.792758 26579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:05.795609 26579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "57ebb6de730c48aca75baf4ed99ecbf0"
format_stamp: "Formatted at 2026-09-27 13:56:05 on dist-test-slave-gkks"
server_key: "5cb908d6067a6ac64694ed850681326c"
server_key_iv: "e0af3b863d0941ff7f6d2320636d46c0"
server_key_version: "encryptionkey@0"
I20260927 13:56:05.796370 26579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "57ebb6de730c48aca75baf4ed99ecbf0"
format_stamp: "Formatted at 2026-09-27 13:56:05 on dist-test-slave-gkks"
server_key: "5cb908d6067a6ac64694ed850681326c"
server_key_iv: "e0af3b863d0941ff7f6d2320636d46c0"
server_key_version: "encryptionkey@0"
I20260927 13:56:05.801119 26579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 13:56:05.804311 26594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:05.805516 26579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:56:05.805685 26579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "57ebb6de730c48aca75baf4ed99ecbf0"
format_stamp: "Formatted at 2026-09-27 13:56:05 on dist-test-slave-gkks"
server_key: "5cb908d6067a6ac64694ed850681326c"
server_key_iv: "e0af3b863d0941ff7f6d2320636d46c0"
server_key_version: "encryptionkey@0"
I20260927 13:56:05.805883 26579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:05.829113 26579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:05.832921 26579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:05.833220 26579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:05.834120 26579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:05.835453 26579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:05.835556 26579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:05.835703 26579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:05.835776 26579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:05.868008 26579 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:46203
I20260927 13:56:05.868090 26706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:46203 every 8 connection(s)
I20260927 13:56:05.869342 26579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:56:05.873798 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 26579
I20260927 13:56:05.873971 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:56:05.874560 22635 external_mini_cluster.cc:1428] Setting key 769322fc2c5040ec6cbec7af2cab1846
Sep 27 13:56:05 dist-test-slave-gkks krb5kdc[26206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517365, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:05.889402 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:05.872191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:58473 (local address 127.22.26.254:41217)
0927 13:56:05.872395 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:05.872401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:05.873719 (+ 1318us) server_negotiation.cc:407] Connection header received
0927 13:56:05.876000 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:05.876005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:05.876098 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:05.876202 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:05.877818 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.878534 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.880150 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.880564 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.880700 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:05.883914 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:05.883919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:05.883922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:05.883953 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:05.886466 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:05.887166 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:05.887171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:05.887175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:05.887304 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:05.887737 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:05.887742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:05.887747 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:05.888007 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:05.888159 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:05.888819 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 13:56:05.889049 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 13:56:05.890439 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:05.872774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:41217 (local address 127.22.26.195:58473)
0927 13:56:05.873519 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:05.873587 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:05.875723 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:05.876456 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:05.876468 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:05.877100 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:05.877641 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:05.877658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.878690 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.878696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:05.880001 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:05.880014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.880797 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.880804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:05.881038 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:05.881642 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:05.881663 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:05.883692 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:05.886595 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:05.886603 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:05.886617 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:05.887020 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:05.887428 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:05.887433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:05.887437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:05.887615 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:05.888144 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:05.888151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:05.888464 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 13:56:05.888713 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 13:56:05.889201 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":287,"threads_started":1}
I20260927 13:56:05.891906 26707 heartbeater.cc:347] Connected to a master server at 127.22.26.254:41217
I20260927 13:56:05.892189 26707 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:05.892846 26707 heartbeater.cc:511] Master 127.22.26.254:41217 requested a full tablet report, sending...
I20260927 13:56:05.894260 26251 ts_manager.cc:194] Registered new tserver with Master: 57ebb6de730c48aca75baf4ed99ecbf0 (127.22.26.195:46203)
I20260927 13:56:05.895185 26251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:58473
I20260927 13:56:05.902253 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:56:05.927963 26440 heartbeater.cc:503] Master 127.22.26.254:41217 was elected leader, sending a full tablet report...
Sep 27 13:56:05 dist-test-slave-gkks krb5kdc[26206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:56:05 dist-test-slave-gkks krb5kdc[26206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:56:05.950126 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:05.935223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.22.26.193:44993)
0927 13:56:05.935941 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:05.935948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:05.935971 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:56:05.936024 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:05.936030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:05.936245 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:05.936411 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:05.937858 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.938857 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.940147 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.940714 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.940849 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:05.944745 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:05.944755 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:05.944768 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:05.944797 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:05.947454 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:05.948087 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:05.948093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:05.948096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:05.948213 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:05.948599 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:05.948604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:05.948606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:05.948920 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:05.949229 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:05.949527 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 13:56:05.949792 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":324,"threads_started":1}
Sep 27 13:56:05 dist-test-slave-gkks krb5kdc[26206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:05.967237 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:05.954157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.22.26.254:41217)
0927 13:56:05.954390 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:05.954396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:05.954557 (+ 161us) server_negotiation.cc:407] Connection header received
0927 13:56:05.954764 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:05.954769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:05.954844 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:05.954946 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:05.956138 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.956843 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.958113 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.958534 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.958663 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:05.962314 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:05.962320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:05.962323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:05.962352 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:05.964712 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:05.965294 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:05.965300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:05.965304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:05.965437 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:05.965808 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:05.965811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:05.965813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:05.966106 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:05.966300 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:05.966759 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 13:56:05.966931 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 13:56:05.967818 26251 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42686
I20260927 13:56:05.968866 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 26312
I20260927 13:56:05.981491 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:05.969663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.22.26.193:44993)
0927 13:56:05.969910 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:05.969916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:05.970044 (+ 128us) server_negotiation.cc:407] Connection header received
0927 13:56:05.970157 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:05.970160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:05.970219 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:05.970298 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:05.971514 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.972264 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.973406 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:05.973806 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:05.973901 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:05.975967 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:05.975972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:05.975975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:05.976003 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:05.979126 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:05.979988 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:05.979992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:05.979994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:05.980107 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:05.980416 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:05.980419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:05.980421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:05.980641 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:05.980820 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:05.981034 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 13:56:05.981183 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 13:56:05.982046 26435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:06.064734 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 26312
I20260927 13:56:06.080960 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 26446
Sep 27 13:56:06 dist-test-slave-gkks krb5kdc[26206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:56:06.096910 26719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:06.081909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.22.26.194:36393)
0927 13:56:06.082446 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:06.082453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:06.082478 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:56:06.082539 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:06.082545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:06.082761 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:06.082889 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:06.084078 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:06.085046 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:06.086456 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:06.087063 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:06.087309 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:06.091393 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:06.091405 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:06.091418 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:06.091448 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:06.094027 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:06.094590 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:06.094596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:06.094598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:06.094703 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:06.095405 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:06.095409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:06.095411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:06.095725 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:06.095948 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:06.096305 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 13:56:06.096501 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":268,"threads_started":1}
I20260927 13:56:06.098140 26569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:06.180255 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 26446
I20260927 13:56:06.196563 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 26579
Sep 27 13:56:06 dist-test-slave-gkks krb5kdc[26206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:56:06.212980 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:06.197411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.22.26.195:46203)
0927 13:56:06.197994 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:06.198001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:06.198028 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:56:06.198092 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:06.198098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:06.198316 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:06.198494 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:06.199865 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:06.200900 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:06.202152 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:06.202722 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:06.202866 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:06.206955 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:06.206965 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:06.206979 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:06.207014 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:06.210049 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:06.210740 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:06.210748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:06.210751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:06.210881 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:06.211298 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:06.211305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:06.211312 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:06.211648 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:06.211977 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:06.212300 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 13:56:06.212581 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":281,"threads_started":1}
I20260927 13:56:06.214298 26702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:06.298774 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 26579
I20260927 13:56:06.314884 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 26219
I20260927 13:56:06.316226 26281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:06.418463 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 26219
2026-09-27T13:56:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3786 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:56:06 dist-test-slave-gkks krb5kdc[26726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:56:06 dist-test-slave-gkks krb5kdc[26726](info): set up 2 sockets
Sep 27 13:56:06 dist-test-slave-gkks krb5kdc[26726](info): commencing operation
krb5kdc: starting...
W20260927 13:56:07.507856 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:56:07 dist-test-slave-gkks krb5kdc[26726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:56:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:56:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:56:07.698876 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:36931
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:36931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:08.025245 26739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:08.025573 26739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:08.025806 26739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:08.033331 26739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:08.033449 26739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:08.033577 26739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:08.033726 26739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:08.033812 26739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:56:08.044059 26739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:36931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:36931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:08.045396 26739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:08.046778 26739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:08.059130 26745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:08.059170 26744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:08.059202 26747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:08.060092 26739 server_base.cc:1034] running on GCE node
I20260927 13:56:08.060789 26739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:08.062084 26739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:08.063328 26739 hybrid_clock.cc:648] HybridClock initialized: now 1790517368063278 us; error 43 us; skew 500 ppm
I20260927 13:56:08.063644 26739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:08 dist-test-slave-gkks krb5kdc[26726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517368, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:08.071821 26739 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:56:08.072981 26739 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:08.073089 26739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:08.073421 26739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:08.076303 26739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "b4d94f27a2774d4a8b4d09677da29063"
format_stamp: "Formatted at 2026-09-27 13:56:08 on dist-test-slave-gkks"
server_key: "7c7d94d3ea5b684803f674b16e4c0fa8"
server_key_iv: "6f7fb8f547ca38976957a224ad678329"
server_key_version: "encryptionkey@0"
I20260927 13:56:08.077118 26739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "b4d94f27a2774d4a8b4d09677da29063"
format_stamp: "Formatted at 2026-09-27 13:56:08 on dist-test-slave-gkks"
server_key: "7c7d94d3ea5b684803f674b16e4c0fa8"
server_key_iv: "6f7fb8f547ca38976957a224ad678329"
server_key_version: "encryptionkey@0"
I20260927 13:56:08.081981 26739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 13:56:08.085340 26753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:08.086755 26739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:56:08.086967 26739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "b4d94f27a2774d4a8b4d09677da29063"
format_stamp: "Formatted at 2026-09-27 13:56:08 on dist-test-slave-gkks"
server_key: "7c7d94d3ea5b684803f674b16e4c0fa8"
server_key_iv: "6f7fb8f547ca38976957a224ad678329"
server_key_version: "encryptionkey@0"
I20260927 13:56:08.087285 26739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:08.111819 26739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:08.116113 26739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:08.116405 26739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:08.134601 26739 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:36931
I20260927 13:56:08.134624 26804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:36931 every 8 connection(s)
I20260927 13:56:08.135499 26739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:56:08.136317 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 26739
I20260927 13:56:08.136549 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:56:08.137303 22635 external_mini_cluster.cc:1428] Setting key 5657bef9c071426229dc5e9b44662582
I20260927 13:56:08.139797 26805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:08.150053 26805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063: Bootstrap starting.
Sep 27 13:56:08 dist-test-slave-gkks krb5kdc[26726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:08.153710 26805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:08.154767 26805 log.cc:824] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:08.157773 26805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063: No bootstrap required, opened a new log
I20260927 13:56:08.158816 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:08.139335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.22.26.254:36931)
0927 13:56:08.140161 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:08.140172 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:08.140212 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:56:08.141789 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:08.141808 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:08.142181 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:08.142557 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:08.144066 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:08.145356 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:08.146646 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:08.147373 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:08.147573 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:08.151909 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:08.151917 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:08.151929 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:08.151968 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:08.155032 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:08.155834 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:08.155843 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:08.155851 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:08.156030 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:08.156429 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:08.156432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:08.156434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:08.156889 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:08.157106 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:08.157538 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 13:56:08.157853 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":380,"threads_started":1}
I20260927 13:56:08.165650 26805 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d94f27a2774d4a8b4d09677da29063" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36931 } }
I20260927 13:56:08.165983 26805 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:08.166101 26805 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d94f27a2774d4a8b4d09677da29063, State: Initialized, Role: FOLLOWER
I20260927 13:56:08.166626 26805 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d94f27a2774d4a8b4d09677da29063" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36931 } }
I20260927 13:56:08.166857 26805 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:56:08.167013 26805 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:56:08.167214 26805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:08.169418 26805 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d94f27a2774d4a8b4d09677da29063" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36931 } }
I20260927 13:56:08.169899 26805 leader_election.cc:302] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4d94f27a2774d4a8b4d09677da29063; no voters:
I20260927 13:56:08.170449 26805 leader_election.cc:288] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:56:08.170590 26810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:08.171306 26810 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [term 1 LEADER]: Becoming Leader. State: Replica: b4d94f27a2774d4a8b4d09677da29063, State: Running, Role: LEADER
I20260927 13:56:08.171866 26810 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d94f27a2774d4a8b4d09677da29063" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36931 } }
I20260927 13:56:08.172516 26805 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:08.174599 26812 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4d94f27a2774d4a8b4d09677da29063. Latest consensus state: current_term: 1 leader_uuid: "b4d94f27a2774d4a8b4d09677da29063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d94f27a2774d4a8b4d09677da29063" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36931 } } }
I20260927 13:56:08.174655 26811 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4d94f27a2774d4a8b4d09677da29063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d94f27a2774d4a8b4d09677da29063" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36931 } } }
I20260927 13:56:08.175122 26811 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:08.174991 26812 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:08.176106 26815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:08.182286 26815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:08.193646 26815 catalog_manager.cc:1347] Generated new cluster ID: 4eae5c0da0924a0c8727b1594c368506
I20260927 13:56:08.193792 26815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:08.223413 26815 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:56:08.224706 26815 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:08.229903 26815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063: Generated new TSK 0
I20260927 13:56:08.230926 26815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:56:08.285029 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36931
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:08.601240 26831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:08.601560 26831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:08.601727 26831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:08.608804 26831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:08.608946 26831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:08.609113 26831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:56:08.618444 26831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:08.619876 26831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:08.621137 26831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:08.633528 26836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:08.633582 26837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:08.634116 26839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:08.634219 26831 server_base.cc:1034] running on GCE node
I20260927 13:56:08.634833 26831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:08.635845 26831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:08.637035 26831 hybrid_clock.cc:648] HybridClock initialized: now 1790517368636985 us; error 52 us; skew 500 ppm
I20260927 13:56:08.637355 26831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:08 dist-test-slave-gkks krb5kdc[26726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517368, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:08.640380 26831 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:56:08.641412 26831 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:08.641522 26831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:08.641842 26831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:08.644665 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "b187d6d1e3c146f09691d06faea02c5e"
format_stamp: "Formatted at 2026-09-27 13:56:08 on dist-test-slave-gkks"
server_key: "9d48e7307e4b522d844c37bb740ddfe1"
server_key_iv: "501f12138d2509116f74d6b51c91949f"
server_key_version: "encryptionkey@0"
I20260927 13:56:08.645428 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "b187d6d1e3c146f09691d06faea02c5e"
format_stamp: "Formatted at 2026-09-27 13:56:08 on dist-test-slave-gkks"
server_key: "9d48e7307e4b522d844c37bb740ddfe1"
server_key_iv: "501f12138d2509116f74d6b51c91949f"
server_key_version: "encryptionkey@0"
I20260927 13:56:08.650126 26831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 13:56:08.653512 26845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:08.654878 26831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:56:08.655045 26831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "b187d6d1e3c146f09691d06faea02c5e"
format_stamp: "Formatted at 2026-09-27 13:56:08 on dist-test-slave-gkks"
server_key: "9d48e7307e4b522d844c37bb740ddfe1"
server_key_iv: "501f12138d2509116f74d6b51c91949f"
server_key_version: "encryptionkey@0"
I20260927 13:56:08.655292 26831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:08.668511 26831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:08.672442 26831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:08.672730 26831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:08.673733 26831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:08.675072 26831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:08.675242 26831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:08.675349 26831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:08.675441 26831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:08.708716 26831 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:38509
I20260927 13:56:08.708778 26957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:38509 every 8 connection(s)
I20260927 13:56:08.709604 26831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:56:08.714900 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 26831
I20260927 13:56:08.715149 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:56:08.715816 22635 external_mini_cluster.cc:1428] Setting key b762cd1a54617807ae661d915e27f5cb
Sep 27 13:56:08 dist-test-slave-gkks krb5kdc[26726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517368, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:08.729983 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:08.712061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:41687 (local address 127.22.26.254:36931)
0927 13:56:08.712545 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:08.712549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:08.713319 (+ 770us) server_negotiation.cc:407] Connection header received
0927 13:56:08.715228 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:08.715236 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:08.715326 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:08.715396 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:08.717270 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:08.718146 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:08.719868 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:08.720259 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:08.720394 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:08.723889 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:08.723894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:08.723897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:08.723928 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:08.726548 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:08.727504 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:08.727507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:08.727510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:08.727622 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:08.728081 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:08.728084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:08.728086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:08.728362 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:08.728475 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:08.729395 (+ 920us) server_negotiation.cc:299] Negotiation successful
0927 13:56:08.729666 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":274,"threads_started":1}
I20260927 13:56:08.731407 26961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:08.712477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:36931 (local address 127.22.26.193:41687)
0927 13:56:08.713134 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:08.713177 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:08.714848 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:08.715820 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:08.715828 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:08.716456 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:08.717076 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:08.717095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:08.718298 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:08.718304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:08.719709 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:08.719722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:08.720483 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:08.720493 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:08.720729 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:08.721424 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:08.721450 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:08.723667 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:08.726684 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:08.726696 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:08.726715 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:08.727396 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:08.727731 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:08.727739 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:08.727745 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:08.727972 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:08.728499 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:08.728510 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:08.728952 (+ 442us) client_negotiation.cc:769] Sending connection context
0927 13:56:08.729273 (+ 321us) client_negotiation.cc:240] Negotiation successful
0927 13:56:08.729854 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":270,"threads_started":1}
I20260927 13:56:08.733289 26958 heartbeater.cc:347] Connected to a master server at 127.22.26.254:36931
I20260927 13:56:08.733654 26958 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:08.734324 26958 heartbeater.cc:511] Master 127.22.26.254:36931 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
I20260927 13:56:08.736538 26770 ts_manager.cc:194] Registered new tserver with Master: b187d6d1e3c146f09691d06faea02c5e (127.22.26.193:38509)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
I20260927 13:56:08.738052 26770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:41687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:56:08.761961 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36931
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:09.077714 26964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:09.077994 26964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:09.078090 26964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:09.085176 26964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:09.085335 26964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:09.085505 26964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:56:09.095285 26964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:09.096539 26964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:09.097834 26964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:09.110536 26969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:09.110601 26972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:09.110630 26970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:09.111372 26964 server_base.cc:1034] running on GCE node
I20260927 13:56:09.111938 26964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:09.112848 26964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:09.114073 26964 hybrid_clock.cc:648] HybridClock initialized: now 1790517369114039 us; error 46 us; skew 500 ppm
I20260927 13:56:09.114380 26964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:09 dist-test-slave-gkks krb5kdc[26726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517369, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:09.117385 26964 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:56:09.118454 26964 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:09.118541 26964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:09.118794 26964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:09.121603 26964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "51b67d5ac0284add9bdc68a6da307ca1"
format_stamp: "Formatted at 2026-09-27 13:56:09 on dist-test-slave-gkks"
server_key: "de526177f361c6957df06864ba892c97"
server_key_iv: "f4615ce9d6dc56c75b086581a6918af1"
server_key_version: "encryptionkey@0"
I20260927 13:56:09.122334 26964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "51b67d5ac0284add9bdc68a6da307ca1"
format_stamp: "Formatted at 2026-09-27 13:56:09 on dist-test-slave-gkks"
server_key: "de526177f361c6957df06864ba892c97"
server_key_iv: "f4615ce9d6dc56c75b086581a6918af1"
server_key_version: "encryptionkey@0"
I20260927 13:56:09.127071 26964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 13:56:09.130402 26978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:09.131778 26964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:09.131944 26964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "51b67d5ac0284add9bdc68a6da307ca1"
format_stamp: "Formatted at 2026-09-27 13:56:09 on dist-test-slave-gkks"
server_key: "de526177f361c6957df06864ba892c97"
server_key_iv: "f4615ce9d6dc56c75b086581a6918af1"
server_key_version: "encryptionkey@0"
I20260927 13:56:09.132145 26964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:09.148450 26964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:09.152537 26964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:09.152858 26964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:09.153847 26964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:09.155220 26964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:09.155306 26964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:09.155380 26964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:09.155435 26964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:09.189661 26964 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:41337
I20260927 13:56:09.189749 27090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:41337 every 8 connection(s)
I20260927 13:56:09.190531 26964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:56:09.200438 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 26964
I20260927 13:56:09.200644 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:56:09.201150 22635 external_mini_cluster.cc:1428] Setting key f4784b5dd94becbf57da424e90a306bd
Sep 27 13:56:09 dist-test-slave-gkks krb5kdc[26726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517369, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:09.210491 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.193077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:57925 (local address 127.22.26.254:36931)
0927 13:56:09.193246 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:09.193250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:09.194412 (+ 1162us) server_negotiation.cc:407] Connection header received
0927 13:56:09.196230 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:09.196238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:09.196327 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:09.196447 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:09.197921 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.198657 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.200485 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.201003 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.201109 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.204826 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:09.204830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:09.204833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:09.204865 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:09.207494 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:09.208216 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:09.208226 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:09.208228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:09.208369 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:09.208818 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:09.208822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:09.208824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:09.209079 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:09.209237 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:09.209928 (+ 691us) server_negotiation.cc:299] Negotiation successful
0927 13:56:09.210159 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:56:09.211598 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.193590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:36931 (local address 127.22.26.194:57925)
0927 13:56:09.194250 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:09.194298 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:09.195967 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:09.196670 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:09.196678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:09.197179 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:09.197745 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.197767 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.198813 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.198818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.200317 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.200333 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.201236 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.201245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.201545 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.202306 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:09.202328 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:09.204605 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:09.207635 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:09.207644 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:09.207662 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:09.208082 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:09.208488 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:09.208494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:09.208497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:09.208680 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:09.209211 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:09.209219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:09.209569 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 13:56:09.209834 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 13:56:09.210376 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":263,"threads_started":1}
I20260927 13:56:09.213055 27091 heartbeater.cc:347] Connected to a master server at 127.22.26.254:36931
I20260927 13:56:09.213397 27091 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:09.214169 27091 heartbeater.cc:511] Master 127.22.26.254:36931 requested a full tablet report, sending...
I20260927 13:56:09.215828 26770 ts_manager.cc:194] Registered new tserver with Master: 51b67d5ac0284add9bdc68a6da307ca1 (127.22.26.194:41337)
I20260927 13:56:09.217150 26770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:57925
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:56:09.246264 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36931
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:09.572046 27096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:09.572331 27096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:09.572433 27096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:09.580462 27096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:09.580564 27096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:09.580694 27096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:56:09.590867 27096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:09.592296 27096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:09.593565 27096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:09.606365 27101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:09.606333 27104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:09.606333 27102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:09.606957 27096 server_base.cc:1034] running on GCE node
I20260927 13:56:09.607570 27096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:09.608486 27096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:09.609735 27096 hybrid_clock.cc:648] HybridClock initialized: now 1790517369609692 us; error 47 us; skew 500 ppm
I20260927 13:56:09.610064 27096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:09 dist-test-slave-gkks krb5kdc[26726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517369, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:09.613395 27096 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:56:09.614485 27096 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:09.614652 27096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:09.614964 27096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:09.617985 27096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "c93814e41c644b5badb0ea4d4bf55082"
format_stamp: "Formatted at 2026-09-27 13:56:09 on dist-test-slave-gkks"
server_key: "796478af5dbea5942a04cf2b098e6d69"
server_key_iv: "0e15a8dac97617fa72a2379b27877709"
server_key_version: "encryptionkey@0"
I20260927 13:56:09.618821 27096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "c93814e41c644b5badb0ea4d4bf55082"
format_stamp: "Formatted at 2026-09-27 13:56:09 on dist-test-slave-gkks"
server_key: "796478af5dbea5942a04cf2b098e6d69"
server_key_iv: "0e15a8dac97617fa72a2379b27877709"
server_key_version: "encryptionkey@0"
I20260927 13:56:09.623971 27096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 13:56:09.627351 27110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:09.628736 27096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:56:09.628922 27096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "c93814e41c644b5badb0ea4d4bf55082"
format_stamp: "Formatted at 2026-09-27 13:56:09 on dist-test-slave-gkks"
server_key: "796478af5dbea5942a04cf2b098e6d69"
server_key_iv: "0e15a8dac97617fa72a2379b27877709"
server_key_version: "encryptionkey@0"
I20260927 13:56:09.629137 27096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:09.640365 27096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:09.644474 27096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:09.644754 27096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:09.645630 27096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:09.646924 27096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:09.647051 27096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:09.647233 27096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:09.647336 27096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:09.679817 27096 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:44733
I20260927 13:56:09.679901 27222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:44733 every 8 connection(s)
I20260927 13:56:09.680646 27096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:56:09.684490 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 27096
I20260927 13:56:09.684691 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:56:09.685263 22635 external_mini_cluster.cc:1428] Setting key 534e528577948fbe002ee50123a44743
Sep 27 13:56:09 dist-test-slave-gkks krb5kdc[26726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517369, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:09.701820 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.682997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:56487 (local address 127.22.26.254:36931)
0927 13:56:09.683199 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:09.683203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:09.684420 (+ 1217us) server_negotiation.cc:407] Connection header received
0927 13:56:09.686776 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:09.686779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:09.686845 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:09.686919 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:09.688889 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.689752 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.691425 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.691869 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.691970 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.695841 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:09.695845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:09.695848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:09.695879 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:09.698571 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:09.699495 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:09.699500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:09.699510 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:09.699688 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:09.700121 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:09.700126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:09.700129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:09.700436 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:09.700598 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:09.701269 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 13:56:09.701486 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:56:09.702724 27225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.683484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:36931 (local address 127.22.26.195:56487)
0927 13:56:09.684215 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:09.684272 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:09.686457 (+ 2185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:09.687256 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:09.687269 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:09.687894 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:09.688673 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.688702 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.689888 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.689892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.691270 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.691286 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.692115 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.692122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.692318 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.692989 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:09.693017 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:09.695626 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:09.698725 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:09.698733 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:09.698747 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:09.699367 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:09.699815 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:09.699820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:09.699824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:09.699996 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:09.700569 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:09.700577 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:09.700918 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 13:56:09.701173 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 13:56:09.701648 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":266,"threads_started":1}
I20260927 13:56:09.704016 27223 heartbeater.cc:347] Connected to a master server at 127.22.26.254:36931
I20260927 13:56:09.704314 27223 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:09.704941 27223 heartbeater.cc:511] Master 127.22.26.254:36931 requested a full tablet report, sending...
I20260927 13:56:09.706323 26770 ts_manager.cc:194] Registered new tserver with Master: c93814e41c644b5badb0ea4d4bf55082 (127.22.26.195:44733)
I20260927 13:56:09.707151 26770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:56487
I20260927 13:56:09.712749 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:56:09.729087 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.717078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.22.26.254:36931)
0927 13:56:09.717283 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:09.717291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:09.717789 (+ 498us) server_negotiation.cc:407] Connection header received
0927 13:56:09.717908 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:09.717911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:09.717968 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:09.718039 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:09.719418 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.720112 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.721396 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.721818 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.721911 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.724115 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:09.724120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:09.724134 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:09.724163 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:09.726760 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:09.727407 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:09.727411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:09.727413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:09.727538 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:09.727859 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:09.727862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:09.727864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:09.728146 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:09.728343 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:09.728606 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 13:56:09.728758 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 13:56:09.732841 26770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:56:09.739648 26770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:56:09.741847 26958 heartbeater.cc:503] Master 127.22.26.254:36931 was elected leader, sending a full tablet report...
I20260927 13:56:09.800877 27237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.751465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.22.26.193:38509)
0927 13:56:09.789670 (+ 38205us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:09.789681 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:09.789720 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:56:09.789872 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:09.789883 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:09.790234 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:09.790495 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:09.792769 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.794489 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.797310 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.798748 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.798895 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.799169 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:09.800337 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0927 13:56:09.800504 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37990,"thread_start_us":40326,"threads_started":1}
I20260927 13:56:09.801743 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.751411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:38509 (local address 127.0.0.1:55814)
0927 13:56:09.786305 (+ 34894us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:09.786375 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:09.786681 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:09.790895 (+ 4214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:09.790908 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:09.790956 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:09.791538 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.791555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.794647 (+ 3092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.794660 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.797127 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.797141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.798979 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.798985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.799481 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.799550 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 13:56:09.799671 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 13:56:09.801267 (+ 1596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34471,"mutex_wait_us":6,"thread_start_us":42677,"threads_started":1}
I20260927 13:56:09.801853 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.752081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56980 (local address 127.22.26.195:44733)
0927 13:56:09.793966 (+ 41885us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:09.793974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:09.794004 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:56:09.794089 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:09.794096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:09.794305 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:09.794495 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:09.795379 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.796790 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.799275 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.800715 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.800839 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.801256 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:09.801341 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 13:56:09.801508 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41714,"thread_start_us":41553,"threads_started":1}
I20260927 13:56:09.801790 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.751441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:44733 (local address 127.0.0.1:56980)
0927 13:56:09.792460 (+ 41019us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:09.792477 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:09.792580 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:09.794571 (+ 1991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:09.794575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:09.794600 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:09.795116 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.795130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.796917 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.796925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.799065 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.799102 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.800912 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.800918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.801066 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.801095 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 13:56:09.801161 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 13:56:09.801291 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40876,"mutex_wait_us":52,"thread_start_us":40711,"threads_started":1}
I20260927 13:56:09.803277 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.752413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:41337 (local address 127.0.0.1:36432)
0927 13:56:09.794347 (+ 41934us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:09.794362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:09.794508 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:09.795240 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:09.795245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:09.795269 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:09.795959 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.795974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.797976 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.797983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.799998 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.800016 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.802546 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.802552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.802707 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.802737 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:56:09.802818 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 13:56:09.802939 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41806,"mutex_wait_us":242,"thread_start_us":34703,"threads_started":1}
I20260927 13:56:09.804059 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.753066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.22.26.194:41337)
0927 13:56:09.788821 (+ 35755us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:09.788834 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:09.794480 (+ 5646us) server_negotiation.cc:407] Connection header received
0927 13:56:09.794708 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:09.794719 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:09.794962 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:09.795165 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:09.796486 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.797839 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.800190 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.801800 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.802214 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.802411 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:09.803593 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0927 13:56:09.803751 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35470,"thread_start_us":35773,"threads_started":1}
I20260927 13:56:09.805769 27158 tablet_service.cc:1467] Processing CreateTablet for tablet 39a0076018d14db6bc05331ec1247915 (DEFAULT_TABLE table=test-table [id=59f0f15856c74fa1a370069d43bcfa0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:09.805778 26893 tablet_service.cc:1467] Processing CreateTablet for tablet 39a0076018d14db6bc05331ec1247915 (DEFAULT_TABLE table=test-table [id=59f0f15856c74fa1a370069d43bcfa0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:09.807291 27158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a0076018d14db6bc05331ec1247915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:09.807282 26893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a0076018d14db6bc05331ec1247915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:09.807195 27026 tablet_service.cc:1467] Processing CreateTablet for tablet 39a0076018d14db6bc05331ec1247915 (DEFAULT_TABLE table=test-table [id=59f0f15856c74fa1a370069d43bcfa0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:09.808413 27026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a0076018d14db6bc05331ec1247915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:09.819043 27241 tablet_bootstrap.cc:492] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1: Bootstrap starting.
I20260927 13:56:09.821707 27242 tablet_bootstrap.cc:492] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082: Bootstrap starting.
I20260927 13:56:09.822361 27243 tablet_bootstrap.cc:492] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e: Bootstrap starting.
I20260927 13:56:09.824118 27241 tablet_bootstrap.cc:654] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:09.825397 27241 log.cc:824] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:09.825536 27243 tablet_bootstrap.cc:654] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:09.825789 27242 tablet_bootstrap.cc:654] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:09.826678 27243 log.cc:824] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:09.826855 27242 log.cc:824] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:09.828532 27241 tablet_bootstrap.cc:492] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1: No bootstrap required, opened a new log
I20260927 13:56:09.828800 27241 ts_tablet_manager.cc:1397] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 13:56:09.829564 27242 tablet_bootstrap.cc:492] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082: No bootstrap required, opened a new log
I20260927 13:56:09.829831 27242 ts_tablet_manager.cc:1397] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260927 13:56:09.829877 27243 tablet_bootstrap.cc:492] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e: No bootstrap required, opened a new log
I20260927 13:56:09.830120 27243 ts_tablet_manager.cc:1397] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260927 13:56:09.837471 27242 raft_consensus.cc:348] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b187d6d1e3c146f09691d06faea02c5e" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38509 } } peers { permanent_uuid: "c93814e41c644b5badb0ea4d4bf55082" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44733 } } peers { permanent_uuid: "51b67d5ac0284add9bdc68a6da307ca1" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 41337 } }
I20260927 13:56:09.837827 27242 raft_consensus.cc:374] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:09.837954 27242 raft_consensus.cc:729] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93814e41c644b5badb0ea4d4bf55082, State: Initialized, Role: FOLLOWER
I20260927 13:56:09.838554 27242 consensus_queue.cc:260] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b187d6d1e3c146f09691d06faea02c5e" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38509 } } peers { permanent_uuid: "c93814e41c644b5badb0ea4d4bf55082" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44733 } } peers { permanent_uuid: "51b67d5ac0284add9bdc68a6da307ca1" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 41337 } }
I20260927 13:56:09.839875 27241 raft_consensus.cc:348] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b187d6d1e3c146f09691d06faea02c5e" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38509 } } peers { permanent_uuid: "c93814e41c644b5badb0ea4d4bf55082" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44733 } } peers { permanent_uuid: "51b67d5ac0284add9bdc68a6da307ca1" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 41337 } }
I20260927 13:56:09.840370 27241 raft_consensus.cc:374] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:09.840523 27241 raft_consensus.cc:729] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b67d5ac0284add9bdc68a6da307ca1, State: Initialized, Role: FOLLOWER
I20260927 13:56:09.840428 27243 raft_consensus.cc:348] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b187d6d1e3c146f09691d06faea02c5e" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38509 } } peers { permanent_uuid: "c93814e41c644b5badb0ea4d4bf55082" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44733 } } peers { permanent_uuid: "51b67d5ac0284add9bdc68a6da307ca1" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 41337 } }
I20260927 13:56:09.840888 27243 raft_consensus.cc:374] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:09.841022 27243 raft_consensus.cc:729] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b187d6d1e3c146f09691d06faea02c5e, State: Initialized, Role: FOLLOWER
I20260927 13:56:09.841241 27241 consensus_queue.cc:260] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b187d6d1e3c146f09691d06faea02c5e" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38509 } } peers { permanent_uuid: "c93814e41c644b5badb0ea4d4bf55082" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44733 } } peers { permanent_uuid: "51b67d5ac0284add9bdc68a6da307ca1" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 41337 } }
I20260927 13:56:09.841720 27243 consensus_queue.cc:260] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b187d6d1e3c146f09691d06faea02c5e" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38509 } } peers { permanent_uuid: "c93814e41c644b5badb0ea4d4bf55082" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44733 } } peers { permanent_uuid: "51b67d5ac0284add9bdc68a6da307ca1" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 41337 } }
I20260927 13:56:09.846014 27223 heartbeater.cc:503] Master 127.22.26.254:36931 was elected leader, sending a full tablet report...
I20260927 13:56:09.846539 27242 ts_tablet_manager.cc:1428] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260927 13:56:09.847604 27243 ts_tablet_manager.cc:1428] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260927 13:56:09.848384 27241 ts_tablet_manager.cc:1428] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1: Time spent starting tablet: real 0.019s user 0.014s sys 0.006s
I20260927 13:56:09.848953 27091 heartbeater.cc:503] Master 127.22.26.254:36931 was elected leader, sending a full tablet report...
I20260927 13:56:09.888526 27248 raft_consensus.cc:482] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:56:09.888856 27248 raft_consensus.cc:504] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b187d6d1e3c146f09691d06faea02c5e" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38509 } } peers { permanent_uuid: "c93814e41c644b5badb0ea4d4bf55082" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44733 } } peers { permanent_uuid: "51b67d5ac0284add9bdc68a6da307ca1" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 41337 } }
I20260927 13:56:09.890946 27248 leader_election.cc:288] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b187d6d1e3c146f09691d06faea02c5e (127.22.26.193:38509), c93814e41c644b5badb0ea4d4bf55082 (127.22.26.195:44733)
I20260927 13:56:09.900260 27237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.891303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:45147 (local address 127.22.26.193:38509)
0927 13:56:09.891571 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:09.891580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:09.892434 (+ 854us) server_negotiation.cc:407] Connection header received
0927 13:56:09.892744 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:09.892749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:09.892825 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:09.892926 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:09.893800 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.894955 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.897908 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.899131 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.899297 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.899372 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:09.899873 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 13:56:09.900004 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 13:56:09.900256 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.891719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:45953 (local address 127.22.26.195:44733)
0927 13:56:09.891969 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:09.891978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:09.892623 (+ 645us) server_negotiation.cc:407] Connection header received
0927 13:56:09.892771 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:09.892776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:09.892844 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:09.892942 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:09.893800 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.894956 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.897900 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.899134 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.899298 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.899372 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:09.899873 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 13:56:09.900004 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 13:56:09.900574 27251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.891281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:44733 (local address 127.22.26.194:45953)
0927 13:56:09.892510 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:09.892529 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:09.892640 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:09.892984 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:09.892989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:09.893032 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:09.893571 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.893592 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.895119 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.895124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.897728 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.897742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.899391 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.899398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.899644 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.899732 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 13:56:09.899831 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 13:56:09.900044 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":250,"spinlock_wait_cycles":768,"thread_start_us":920,"threads_started":1}
I20260927 13:56:09.900596 27250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.891143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:38509 (local address 127.22.26.194:45147)
0927 13:56:09.892329 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:09.892346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:09.892517 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:09.892984 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:09.892989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:09.893040 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:09.893650 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.893659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.895118 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.895124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.897731 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:09.897742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.899404 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.899410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:09.899653 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.899738 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 13:56:09.899831 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 13:56:09.900044 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":279,"spinlock_wait_cycles":22784,"thread_start_us":686,"threads_started":1}
I20260927 13:56:09.901618 26913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a0076018d14db6bc05331ec1247915" candidate_uuid: "51b67d5ac0284add9bdc68a6da307ca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b187d6d1e3c146f09691d06faea02c5e" is_pre_election: true
I20260927 13:56:09.901623 27178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a0076018d14db6bc05331ec1247915" candidate_uuid: "51b67d5ac0284add9bdc68a6da307ca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c93814e41c644b5badb0ea4d4bf55082" is_pre_election: true
I20260927 13:56:09.902122 27178 raft_consensus.cc:2454] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b67d5ac0284add9bdc68a6da307ca1 in term 0.
I20260927 13:56:09.902120 26913 raft_consensus.cc:2454] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b67d5ac0284add9bdc68a6da307ca1 in term 0.
I20260927 13:56:09.902770 26979 leader_election.cc:302] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 51b67d5ac0284add9bdc68a6da307ca1, b187d6d1e3c146f09691d06faea02c5e, c93814e41c644b5badb0ea4d4bf55082; no voters:
I20260927 13:56:09.903220 27248 raft_consensus.cc:2781] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:56:09.903371 27248 raft_consensus.cc:482] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:56:09.903476 27248 raft_consensus.cc:3037] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:09.905826 27248 raft_consensus.cc:504] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b187d6d1e3c146f09691d06faea02c5e" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38509 } } peers { permanent_uuid: "c93814e41c644b5badb0ea4d4bf55082" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44733 } } peers { permanent_uuid: "51b67d5ac0284add9bdc68a6da307ca1" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 41337 } }
I20260927 13:56:09.906454 27248 leader_election.cc:288] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [CANDIDATE]: Term 1 election: Requested vote from peers b187d6d1e3c146f09691d06faea02c5e (127.22.26.193:38509), c93814e41c644b5badb0ea4d4bf55082 (127.22.26.195:44733)
I20260927 13:56:09.906931 26913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a0076018d14db6bc05331ec1247915" candidate_uuid: "51b67d5ac0284add9bdc68a6da307ca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b187d6d1e3c146f09691d06faea02c5e"
I20260927 13:56:09.906977 27178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a0076018d14db6bc05331ec1247915" candidate_uuid: "51b67d5ac0284add9bdc68a6da307ca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c93814e41c644b5badb0ea4d4bf55082"
I20260927 13:56:09.907172 26913 raft_consensus.cc:3037] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:09.907202 27178 raft_consensus.cc:3037] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:09.909343 26913 raft_consensus.cc:2454] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b67d5ac0284add9bdc68a6da307ca1 in term 1.
I20260927 13:56:09.909456 27178 raft_consensus.cc:2454] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b67d5ac0284add9bdc68a6da307ca1 in term 1.
I20260927 13:56:09.909775 26979 leader_election.cc:302] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b67d5ac0284add9bdc68a6da307ca1, b187d6d1e3c146f09691d06faea02c5e; no voters:
I20260927 13:56:09.910059 27248 raft_consensus.cc:2781] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:09.910490 27248 raft_consensus.cc:686] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [term 1 LEADER]: Becoming Leader. State: Replica: 51b67d5ac0284add9bdc68a6da307ca1, State: Running, Role: LEADER
I20260927 13:56:09.910998 27248 consensus_queue.cc:237] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b187d6d1e3c146f09691d06faea02c5e" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38509 } } peers { permanent_uuid: "c93814e41c644b5badb0ea4d4bf55082" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44733 } } peers { permanent_uuid: "51b67d5ac0284add9bdc68a6da307ca1" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 41337 } }
I20260927 13:56:09.916154 26770 catalog_manager.cc:5495] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b67d5ac0284add9bdc68a6da307ca1 (127.22.26.194). New cstate: current_term: 1 leader_uuid: "51b67d5ac0284add9bdc68a6da307ca1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b187d6d1e3c146f09691d06faea02c5e" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c93814e41c644b5badb0ea4d4bf55082" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b67d5ac0284add9bdc68a6da307ca1" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 41337 } health_report { overall_health: HEALTHY } } }
W20260927 13:56:09.932092 27224 tablet.cc:2369] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:56:09.942467 27092 tablet.cc:2369] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:56:09.955356 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:09.950356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.22.26.194:41337)
0927 13:56:09.950610 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:09.950614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:09.950633 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:56:09.950798 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:09.950802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:09.950883 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:09.951001 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:56:09.951403 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.952085 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.953402 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:09.953804 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:09.953898 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:09.954299 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:56:09.954758 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:56:09.954860 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:09.954954 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 13:56:09.955012 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 13:56:09.961719 26959 tablet.cc:2369] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:56:09.963127 26913 raft_consensus.cc:1261] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e [term 1 FOLLOWER]: Refusing update from remote peer 51b67d5ac0284add9bdc68a6da307ca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:09.963126 27178 raft_consensus.cc:1261] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082 [term 1 FOLLOWER]: Refusing update from remote peer 51b67d5ac0284add9bdc68a6da307ca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:09.964164 27252 consensus_queue.cc:1035] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b187d6d1e3c146f09691d06faea02c5e" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:09.964521 27248 consensus_queue.cc:1035] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c93814e41c644b5badb0ea4d4bf55082" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:09.976939 27256 mvcc.cc:205] Tried to move back new op lower bound from 7333959147359260672 to 7333959147160363008. Current Snapshot: MvccSnapshot[applied={T|T < 7333959147359260672}]
I20260927 13:56:09.997608 26769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:09.997900 26769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:10.001626 26769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4d94f27a2774d4a8b4d09677da29063: Sending DeleteTablet for 3 replicas of tablet 39a0076018d14db6bc05331ec1247915
I20260927 13:56:10.002733 27158 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a0076018d14db6bc05331ec1247915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:09 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:56980
I20260927 13:56:10.002767 27026 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a0076018d14db6bc05331ec1247915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:09 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:36432
I20260927 13:56:10.003506 26893 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a0076018d14db6bc05331ec1247915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:09 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:55814
I20260927 13:56:10.004004 27266 tablet_replica.cc:330] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082: stopping tablet replica
I20260927 13:56:10.004419 27266 raft_consensus.cc:2229] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:10.004910 27266 raft_consensus.cc:2258] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:10.005638 27265 tablet_replica.cc:330] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1: stopping tablet replica
I20260927 13:56:10.005872 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 26831
I20260927 13:56:10.006062 27265 raft_consensus.cc:2229] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:56:10.006733 27265 raft_consensus.cc:2258] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:10.007280 27266 ts_tablet_manager.cc:1905] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:10.008430 27267 tablet_replica.cc:330] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e: stopping tablet replica
I20260927 13:56:10.008872 27267 raft_consensus.cc:2229] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:10.009426 27267 raft_consensus.cc:2258] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:10.011499 27265 ts_tablet_manager.cc:1905] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:10.013641 27267 ts_tablet_manager.cc:1905] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:10.015581 27266 ts_tablet_manager.cc:1918] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:10.015813 27266 log.cc:1196] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/39a0076018d14db6bc05331ec1247915
I20260927 13:56:10.016417 27266 ts_tablet_manager.cc:1939] T 39a0076018d14db6bc05331ec1247915 P c93814e41c644b5badb0ea4d4bf55082: Deleting consensus metadata
I20260927 13:56:10.018358 26756 catalog_manager.cc:4841] TS c93814e41c644b5badb0ea4d4bf55082 (127.22.26.195:44733): tablet 39a0076018d14db6bc05331ec1247915 (table test-table [id=59f0f15856c74fa1a370069d43bcfa0e]) successfully deleted
I20260927 13:56:10.020644 27267 ts_tablet_manager.cc:1918] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:10.020905 27267 log.cc:1196] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/39a0076018d14db6bc05331ec1247915
I20260927 13:56:10.021631 27267 ts_tablet_manager.cc:1939] T 39a0076018d14db6bc05331ec1247915 P b187d6d1e3c146f09691d06faea02c5e: Deleting consensus metadata
I20260927 13:56:10.021631 27265 ts_tablet_manager.cc:1918] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:10.021792 27265 log.cc:1196] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/39a0076018d14db6bc05331ec1247915
I20260927 13:56:10.022461 27265 ts_tablet_manager.cc:1939] T 39a0076018d14db6bc05331ec1247915 P 51b67d5ac0284add9bdc68a6da307ca1: Deleting consensus metadata
I20260927 13:56:10.023568 26754 catalog_manager.cc:4841] TS b187d6d1e3c146f09691d06faea02c5e (127.22.26.193:38509): tablet 39a0076018d14db6bc05331ec1247915 (table test-table [id=59f0f15856c74fa1a370069d43bcfa0e]) successfully deleted
I20260927 13:56:10.024196 26757 catalog_manager.cc:4841] TS 51b67d5ac0284add9bdc68a6da307ca1 (127.22.26.194:41337): tablet 39a0076018d14db6bc05331ec1247915 (table test-table [id=59f0f15856c74fa1a370069d43bcfa0e]) successfully deleted
Sep 27 13:56:10 dist-test-slave-gkks krb5kdc[26726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:56:10.034745 27237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:10.007379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.22.26.193:38509)
0927 13:56:10.007555 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:10.007561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:10.012052 (+ 4491us) server_negotiation.cc:407] Connection header received
0927 13:56:10.012362 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:10.012369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:10.012455 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:10.012532 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:10.018934 (+ 6402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:10.020138 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:10.022144 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:10.022785 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:10.022884 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:10.027585 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:10.027599 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:10.027617 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:10.027665 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:10.031525 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:10.032266 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:10.032276 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:10.032280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:10.032445 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:10.032932 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:10.032937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:10.032941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:10.033432 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:10.033737 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:10.034120 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 13:56:10.034356 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 13:56:10.035583 26953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:10.123926 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 26831
I20260927 13:56:10.144443 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 26964
Sep 27 13:56:10 dist-test-slave-gkks krb5kdc[26726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:56:10.163462 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:10.145450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.22.26.194:41337)
0927 13:56:10.145629 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:10.145635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:10.145654 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:56:10.145733 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:10.145738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:10.145827 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:10.145945 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:10.147467 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:10.148734 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:10.150600 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:10.151361 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:10.151487 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:10.156621 (+ 5134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:10.156642 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:10.156661 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:10.156705 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:10.160631 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:10.161417 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:10.161426 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:10.161430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:10.161591 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:10.162007 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:10.162015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:10.162021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:10.162403 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:10.162650 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:10.162927 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 13:56:10.163076 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:56:10.164120 27086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:10.279599 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 26964
I20260927 13:56:10.299489 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 27096
Sep 27 13:56:10 dist-test-slave-gkks krb5kdc[26726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:56:10.314215 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:10.300217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56986 (local address 127.22.26.195:44733)
0927 13:56:10.300402 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:10.300410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:10.300432 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:56:10.300641 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:10.300646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:10.300717 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:10.300834 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:10.302089 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:10.302830 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:10.304150 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:10.304540 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:10.304691 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:10.308874 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:10.308888 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:10.308904 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:10.308935 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:10.311570 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:10.312196 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:10.312204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:10.312206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:10.312340 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:10.312721 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:10.312727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:10.312730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:10.313074 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:10.313362 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:10.313676 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 13:56:10.313918 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:56:10.314869 27218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:10.402833 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 27096
I20260927 13:56:10.418977 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 26739
I20260927 13:56:10.420256 26800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:10.529711 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 26739
2026-09-27T13:56:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4111 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T13:56:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:56:10Z Disabled control of system clock
I20260927 13:56:10.594554 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:43281
--webserver_interface=127.22.26.254
--webserver_port=0
--builtin_ntp_servers=127.22.26.212:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 13:56:10.907829 27276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:10.908105 27276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:10.908190 27276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:10.915274 27276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:10.915385 27276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:10.915422 27276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:10.915472 27276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:10.915515 27276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:56:10.924479 27276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:43281
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:43281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:10.925706 27276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:10.927428 27276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:10.939410 27281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:10.939549 27284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:10.939531 27282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:10.944792 27276 server_base.cc:1034] running on GCE node
I20260927 13:56:10.945653 27276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:10.947146 27276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:10.948364 27276 hybrid_clock.cc:648] HybridClock initialized: now 1790517370948300 us; error 58 us; skew 500 ppm
I20260927 13:56:10.948734 27276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:56:10.951237 27276 webserver.cc:460] Webserver started at http://127.22.26.254:43923/ using document root <none> and password file <none>
I20260927 13:56:10.952051 27276 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:10.952140 27276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:10.952529 27276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:10.955499 27276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "de7aebf51dbe4d82a2bfc2241e25e8ea"
format_stamp: "Formatted at 2026-09-27 13:56:10 on dist-test-slave-gkks"
server_key: "85372975b77ca214ad736230509444a9"
server_key_iv: "b06fa0976fd8f8db3d35e4f50670fe47"
server_key_version: "encryptionkey@0"
I20260927 13:56:10.956308 27276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "de7aebf51dbe4d82a2bfc2241e25e8ea"
format_stamp: "Formatted at 2026-09-27 13:56:10 on dist-test-slave-gkks"
server_key: "85372975b77ca214ad736230509444a9"
server_key_iv: "b06fa0976fd8f8db3d35e4f50670fe47"
server_key_version: "encryptionkey@0"
I20260927 13:56:10.961552 27276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 13:56:10.964906 27290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:10.966220 27276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:56:10.966411 27276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "de7aebf51dbe4d82a2bfc2241e25e8ea"
format_stamp: "Formatted at 2026-09-27 13:56:10 on dist-test-slave-gkks"
server_key: "85372975b77ca214ad736230509444a9"
server_key_iv: "b06fa0976fd8f8db3d35e4f50670fe47"
server_key_version: "encryptionkey@0"
I20260927 13:56:10.966634 27276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:10.972427 27276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:10.973162 27276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:10.973462 27276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:10.991930 27276 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:43281
I20260927 13:56:10.991981 27341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:43281 every 8 connection(s)
I20260927 13:56:10.993275 27276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:56:10.996951 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:11.000679 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 27276
I20260927 13:56:11.000921 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:56:11.001811 22635 external_mini_cluster.cc:1428] Setting key af1d035f9d56883e8759481a7abe6e83
I20260927 13:56:11.010030 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:11.004069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.22.26.254:43281)
0927 13:56:11.005053 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:11.005064 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:11.005104 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:56:11.006607 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:11.006627 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:11.007043 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:11.007434 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:11.008505 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:11.008511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:11.008523 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:11.008540 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:11.008601 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:11.008616 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:11.008797 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:11.008894 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 13:56:11.009142 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":385,"threads_started":1}
I20260927 13:56:11.010799 27342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea: Bootstrap starting.
I20260927 13:56:11.013989 27342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:11.014919 27342 log.cc:824] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:11.017419 27342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea: No bootstrap required, opened a new log
I20260927 13:56:11.024544 27342 raft_consensus.cc:348] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7aebf51dbe4d82a2bfc2241e25e8ea" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43281 } }
I20260927 13:56:11.024864 27342 raft_consensus.cc:374] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:11.024935 27342 raft_consensus.cc:729] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7aebf51dbe4d82a2bfc2241e25e8ea, State: Initialized, Role: FOLLOWER
I20260927 13:56:11.025391 27342 consensus_queue.cc:260] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7aebf51dbe4d82a2bfc2241e25e8ea" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43281 } }
I20260927 13:56:11.025624 27342 raft_consensus.cc:388] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:56:11.025769 27342 raft_consensus.cc:482] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:56:11.025959 27342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:11.028111 27342 raft_consensus.cc:504] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7aebf51dbe4d82a2bfc2241e25e8ea" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43281 } }
I20260927 13:56:11.028553 27342 leader_election.cc:302] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de7aebf51dbe4d82a2bfc2241e25e8ea; no voters:
I20260927 13:56:11.029049 27342 leader_election.cc:288] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:56:11.029210 27347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:11.029973 27347 raft_consensus.cc:686] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [term 1 LEADER]: Becoming Leader. State: Replica: de7aebf51dbe4d82a2bfc2241e25e8ea, State: Running, Role: LEADER
I20260927 13:56:11.030522 27347 consensus_queue.cc:237] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7aebf51dbe4d82a2bfc2241e25e8ea" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43281 } }
I20260927 13:56:11.031198 27342 sys_catalog.cc:564] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:11.033406 27349 sys_catalog.cc:455] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader de7aebf51dbe4d82a2bfc2241e25e8ea. Latest consensus state: current_term: 1 leader_uuid: "de7aebf51dbe4d82a2bfc2241e25e8ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7aebf51dbe4d82a2bfc2241e25e8ea" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43281 } } }
I20260927 13:56:11.033845 27349 sys_catalog.cc:458] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:11.034871 27355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:11.034729 27348 sys_catalog.cc:455] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de7aebf51dbe4d82a2bfc2241e25e8ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7aebf51dbe4d82a2bfc2241e25e8ea" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43281 } } }
I20260927 13:56:11.036247 27348 sys_catalog.cc:458] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:11.040252 27355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:11.048847 27355 catalog_manager.cc:1347] Generated new cluster ID: c98527d01c024fd2a367f667e229ae19
I20260927 13:56:11.048954 27355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:11.081537 27355 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:56:11.082028 27355 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:11.094683 27355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea: Generated new TSK 0
I20260927 13:56:11.095566 27355 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:56:11.110592 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:43281
--builtin_ntp_servers=127.22.26.212:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 13:56:11.459492 27366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:11.459844 27366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:11.459990 27366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:11.467676 27366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:11.467797 27366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:11.468014 27366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:56:11.478322 27366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:43281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:11.480233 27366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:11.482795 27366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:11.495977 27371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:11.496023 27372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:11.496066 27374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:11.497035 27366 server_base.cc:1034] running on GCE node
I20260927 13:56:11.497624 27366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:11.498493 27366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:11.499716 27366 hybrid_clock.cc:648] HybridClock initialized: now 1790517371499664 us; error 59 us; skew 500 ppm
I20260927 13:56:11.500072 27366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:56:11.502306 27366 webserver.cc:460] Webserver started at http://127.22.26.193:39383/ using document root <none> and password file <none>
I20260927 13:56:11.502960 27366 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:11.503122 27366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:11.503465 27366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:11.506131 27366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "15324440106e4e0fbd903bebfb37514c"
format_stamp: "Formatted at 2026-09-27 13:56:11 on dist-test-slave-gkks"
server_key: "723a215d21fddd333efa3eca4209ef8e"
server_key_iv: "3b0b1838c9b8888848f59a65f04a6575"
server_key_version: "encryptionkey@0"
I20260927 13:56:11.506891 27366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "15324440106e4e0fbd903bebfb37514c"
format_stamp: "Formatted at 2026-09-27 13:56:11 on dist-test-slave-gkks"
server_key: "723a215d21fddd333efa3eca4209ef8e"
server_key_iv: "3b0b1838c9b8888848f59a65f04a6575"
server_key_version: "encryptionkey@0"
I20260927 13:56:11.511785 27366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:56:11.514962 27380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:11.516343 27366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:56:11.516503 27366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "15324440106e4e0fbd903bebfb37514c"
format_stamp: "Formatted at 2026-09-27 13:56:11 on dist-test-slave-gkks"
server_key: "723a215d21fddd333efa3eca4209ef8e"
server_key_iv: "3b0b1838c9b8888848f59a65f04a6575"
server_key_version: "encryptionkey@0"
I20260927 13:56:11.516705 27366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:11.522190 27366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:11.522907 27366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:11.523207 27366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:11.524058 27366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:11.525359 27366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:11.525478 27366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:11.525594 27366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:11.525678 27366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:11.558352 27366 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:41023
I20260927 13:56:11.558403 27492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:41023 every 8 connection(s)
I20260927 13:56:11.559716 27366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:56:11.561345 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 27366
I20260927 13:56:11.561568 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:56:11.562369 22635 external_mini_cluster.cc:1428] Setting key 58100b770bd7f71914d014e06823c5a4
I20260927 13:56:11.565600 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:43281
--builtin_ntp_servers=127.22.26.212:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 13:56:11.569344 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:11.562526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:40829 (local address 127.22.26.254:43281)
0927 13:56:11.563039 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:11.563043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:11.564448 (+ 1405us) server_negotiation.cc:407] Connection header received
0927 13:56:11.567021 (+ 2573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:11.567025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:11.567071 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:11.567216 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:11.568678 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:11.568683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:11.568686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:11.568700 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:11.568741 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:11.568767 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:11.568864 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:11.569078 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 13:56:11.569126 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":242,"threads_started":1}
I20260927 13:56:11.570611 27496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:11.563241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43281 (local address 127.22.26.193:40829)
0927 13:56:11.564298 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:11.564343 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:11.566726 (+ 2383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:11.567473 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:11.567487 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:11.567605 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:11.568335 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:56:11.568340 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:11.568392 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:56:11.568508 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:11.568865 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:11.568878 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:11.568889 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 13:56:11.569151 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 13:56:11.569405 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":333,"threads_started":1}
I20260927 13:56:11.572213 27493 heartbeater.cc:347] Connected to a master server at 127.22.26.254:43281
I20260927 13:56:11.572546 27493 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:11.573290 27493 heartbeater.cc:511] Master 127.22.26.254:43281 requested a full tablet report, sending...
I20260927 13:56:11.575404 27307 ts_manager.cc:194] Registered new tserver with Master: 15324440106e4e0fbd903bebfb37514c (127.22.26.193:41023)
W20260927 13:56:11.884768 27497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:11.885041 27497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:11.885181 27497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:11.892391 27497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:11.892484 27497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:11.892666 27497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:56:11.901525 27497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:43281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:11.902812 27497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:11.904616 27497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:11.917661 27505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:11.917695 27503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:11.917769 27502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:11.918397 27497 server_base.cc:1034] running on GCE node
I20260927 13:56:11.919147 27497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:11.920169 27497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:11.921397 27497 hybrid_clock.cc:648] HybridClock initialized: now 1790517371921339 us; error 57 us; skew 500 ppm
I20260927 13:56:11.921715 27497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:56:11.924036 27497 webserver.cc:460] Webserver started at http://127.22.26.194:35007/ using document root <none> and password file <none>
I20260927 13:56:11.924741 27497 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:11.924881 27497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:11.925249 27497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:11.928194 27497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "4877da551a0547398ee7bc7c746442ba"
format_stamp: "Formatted at 2026-09-27 13:56:11 on dist-test-slave-gkks"
server_key: "f8e2a83cf3f7fafd059a665159a4f36f"
server_key_iv: "0cfe314fe3a45572ab8909561792f2c9"
server_key_version: "encryptionkey@0"
I20260927 13:56:11.928938 27497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "4877da551a0547398ee7bc7c746442ba"
format_stamp: "Formatted at 2026-09-27 13:56:11 on dist-test-slave-gkks"
server_key: "f8e2a83cf3f7fafd059a665159a4f36f"
server_key_iv: "0cfe314fe3a45572ab8909561792f2c9"
server_key_version: "encryptionkey@0"
I20260927 13:56:11.934010 27497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:56:11.937780 27511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:11.939241 27497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:11.939407 27497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "4877da551a0547398ee7bc7c746442ba"
format_stamp: "Formatted at 2026-09-27 13:56:11 on dist-test-slave-gkks"
server_key: "f8e2a83cf3f7fafd059a665159a4f36f"
server_key_iv: "0cfe314fe3a45572ab8909561792f2c9"
server_key_version: "encryptionkey@0"
I20260927 13:56:11.939636 27497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:11.945205 27497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:11.945981 27497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:11.946205 27497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:11.947055 27497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:11.948490 27497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:11.948573 27497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:11.948647 27497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:11.948704 27497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:11.981537 27497 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:38011
I20260927 13:56:11.981592 27623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:38011 every 8 connection(s)
I20260927 13:56:11.982851 27497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:56:11.986321 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 27497
I20260927 13:56:11.986514 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:56:11.986963 22635 external_mini_cluster.cc:1428] Setting key d2c88216d9ddd0d72fb04c7b738ed945
I20260927 13:56:11.989831 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:43281
--builtin_ntp_servers=127.22.26.212:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 13:56:11.991274 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:11.985521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:38115 (local address 127.22.26.254:43281)
0927 13:56:11.985740 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:11.985744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:11.987382 (+ 1638us) server_negotiation.cc:407] Connection header received
0927 13:56:11.989029 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:11.989035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:11.989095 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:11.989202 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:11.990540 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:11.990544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:11.990547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:11.990563 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:11.990596 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:11.990608 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:11.990699 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:11.990941 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 13:56:11.990991 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:56:11.992676 27626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:11.986120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43281 (local address 127.22.26.194:38115)
0927 13:56:11.987195 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:11.987257 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:11.988777 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:11.989402 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:11.989411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:11.989500 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:11.990190 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:56:11.990195 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:11.990245 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:56:11.990358 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:11.990719 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:11.990732 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:11.990744 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 13:56:11.991025 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 13:56:11.991821 (+ 796us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":333,"threads_started":1}
I20260927 13:56:11.993993 27624 heartbeater.cc:347] Connected to a master server at 127.22.26.254:43281
I20260927 13:56:11.994299 27624 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:11.994879 27624 heartbeater.cc:511] Master 127.22.26.254:43281 requested a full tablet report, sending...
I20260927 13:56:11.996376 27307 ts_manager.cc:194] Registered new tserver with Master: 4877da551a0547398ee7bc7c746442ba (127.22.26.194:38011)
W20260927 13:56:12.306973 27627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:12.307307 27627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:12.307466 27627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:12.314672 27627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:12.314764 27627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:12.314931 27627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:56:12.324606 27627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:43281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:12.325826 27627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:12.327576 27627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:12.340955 27632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:12.341033 27633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:12.340976 27635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:12.341832 27627 server_base.cc:1034] running on GCE node
I20260927 13:56:12.342415 27627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:12.343461 27627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:12.344679 27627 hybrid_clock.cc:648] HybridClock initialized: now 1790517372344636 us; error 38 us; skew 500 ppm
I20260927 13:56:12.345009 27627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:56:12.347177 27627 webserver.cc:460] Webserver started at http://127.22.26.195:34785/ using document root <none> and password file <none>
I20260927 13:56:12.347837 27627 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:12.347965 27627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:12.348289 27627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:12.351006 27627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "6d393d7841584de7a9252c063cb8db25"
format_stamp: "Formatted at 2026-09-27 13:56:12 on dist-test-slave-gkks"
server_key: "20e2d81708335c5881497213c852415c"
server_key_iv: "9c239a7c49bd936fdcb8b8a544afbce3"
server_key_version: "encryptionkey@0"
I20260927 13:56:12.351845 27627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "6d393d7841584de7a9252c063cb8db25"
format_stamp: "Formatted at 2026-09-27 13:56:12 on dist-test-slave-gkks"
server_key: "20e2d81708335c5881497213c852415c"
server_key_iv: "9c239a7c49bd936fdcb8b8a544afbce3"
server_key_version: "encryptionkey@0"
I20260927 13:56:12.356570 27627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:56:12.359826 27641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:12.361007 27627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:56:12.361181 27627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "6d393d7841584de7a9252c063cb8db25"
format_stamp: "Formatted at 2026-09-27 13:56:12 on dist-test-slave-gkks"
server_key: "20e2d81708335c5881497213c852415c"
server_key_iv: "9c239a7c49bd936fdcb8b8a544afbce3"
server_key_version: "encryptionkey@0"
I20260927 13:56:12.361385 27627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:12.366707 27627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:12.367477 27627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:12.367741 27627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:12.368602 27627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:12.370072 27627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:12.370183 27627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:12.370312 27627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:12.370396 27627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:12.402045 27627 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:45421
I20260927 13:56:12.402098 27753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:45421 every 8 connection(s)
I20260927 13:56:12.403340 27627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:56:12.409629 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 27627
I20260927 13:56:12.409832 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:56:12.410329 22635 external_mini_cluster.cc:1428] Setting key 0ac8f23d22197672ab635839e2786b76
I20260927 13:56:12.411700 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.405933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:46377 (local address 127.22.26.254:43281)
0927 13:56:12.406118 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:12.406123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:12.407575 (+ 1452us) server_negotiation.cc:407] Connection header received
0927 13:56:12.409298 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:12.409304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:12.409376 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:12.409485 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:12.410903 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:12.410908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:12.410911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:12.410926 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:12.410958 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:12.410970 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:12.411133 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:12.411351 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 13:56:12.411422 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:56:12.412693 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.406459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43281 (local address 127.22.26.195:46377)
0927 13:56:12.407422 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:12.407458 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:12.409029 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:12.409730 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:12.409743 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:12.409849 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:12.410596 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:56:12.410598 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:12.410633 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:56:12.410716 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:12.411110 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:12.411121 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:12.411129 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 13:56:12.411427 (+ 298us) client_negotiation.cc:240] Negotiation successful
0927 13:56:12.411664 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":269,"threads_started":1}
I20260927 13:56:12.413966 27754 heartbeater.cc:347] Connected to a master server at 127.22.26.254:43281
I20260927 13:56:12.414248 27754 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:12.414781 27754 heartbeater.cc:511] Master 127.22.26.254:43281 requested a full tablet report, sending...
I20260927 13:56:12.416093 27307 ts_manager.cc:194] Registered new tserver with Master: 6d393d7841584de7a9252c063cb8db25 (127.22.26.195:45421)
I20260927 13:56:12.425235 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:56:12.432293 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.429641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.22.26.254:43281)
0927 13:56:12.429789 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:12.429793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:12.430300 (+ 507us) server_negotiation.cc:407] Connection header received
0927 13:56:12.430477 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:12.430487 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:12.430535 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:12.430633 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:12.431671 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:12.431677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:12.431682 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:12.431699 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:12.431737 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:12.431751 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:12.431895 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:12.431949 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 13:56:12.432008 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:56:12.435814 27307 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:56:12.442528 27307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:56:12.472391 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.453592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:38011 (local address 127.0.0.1:55868)
0927 13:56:12.468011 (+ 14419us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:12.468036 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:12.468157 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:12.469465 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:12.469477 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:12.469603 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:12.470603 (+ 1000us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:56:12.470609 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:12.470646 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:56:12.470750 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:12.471518 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:12.471533 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:12.471547 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 13:56:12.471749 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 13:56:12.471855 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14232,"thread_start_us":17410,"threads_started":1}
I20260927 13:56:12.473990 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.453776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.22.26.194:38011)
0927 13:56:12.468647 (+ 14871us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:12.468659 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:12.468697 (+ 38us) server_negotiation.cc:407] Connection header received
0927 13:56:12.468784 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:12.468798 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:12.469090 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:12.469294 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:12.471196 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:12.471208 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:12.471226 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:12.471258 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:12.471327 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:12.471359 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:12.472882 (+ 1523us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:12.472977 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 13:56:12.473088 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14668,"thread_start_us":16320,"threads_started":1}
I20260927 13:56:12.474900 27771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.455634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.22.26.193:41023)
0927 13:56:12.466622 (+ 10988us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:12.466634 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:12.471560 (+ 4926us) server_negotiation.cc:407] Connection header received
0927 13:56:12.471637 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:12.471644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:12.471855 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:12.471993 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:12.473769 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:12.473779 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:12.473794 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:12.473825 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:12.473884 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:12.473904 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:12.474033 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:12.474483 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 13:56:12.474576 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10771,"thread_start_us":10617,"threads_started":1}
I20260927 13:56:12.474936 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.455372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:41023 (local address 127.0.0.1:50340)
0927 13:56:12.471327 (+ 15955us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:12.471344 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:12.471438 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:12.472828 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:12.472835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:12.472863 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:12.473515 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:56:12.473519 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:12.473534 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:56:12.473583 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:12.474313 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:12.474318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:12.474322 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 13:56:12.474469 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 13:56:12.474517 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15771,"thread_start_us":6713,"threads_started":1}
I20260927 13:56:12.475641 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.453397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:45421 (local address 127.0.0.1:32860)
0927 13:56:12.462451 (+ 9054us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:12.462486 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:12.462614 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:12.473214 (+ 10600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:12.473219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:12.473241 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:12.473873 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:56:12.473876 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:12.473892 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:56:12.473929 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:12.474521 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:12.474525 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:12.474530 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:56:12.474660 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 13:56:12.474719 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8831,"thread_start_us":14862,"threads_started":1}
I20260927 13:56:12.475888 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.453615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.22.26.195:45421)
0927 13:56:12.472725 (+ 19110us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:12.472741 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:12.472771 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:56:12.472838 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:12.472847 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:12.473052 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:12.473207 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:12.474277 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:12.474284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:12.474295 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:12.474310 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:12.474353 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:12.474366 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:12.475352 (+ 986us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:12.475417 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 13:56:12.475498 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18922,"thread_start_us":18761,"threads_started":1}
I20260927 13:56:12.477878 27559 tablet_service.cc:1467] Processing CreateTablet for tablet 7d265495bde7475c8d06293d5881193a (DEFAULT_TABLE table=test-table [id=0cc5d9e394ba47b1a141d1948fae9a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:12.479242 27559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d265495bde7475c8d06293d5881193a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:12.479207 27428 tablet_service.cc:1467] Processing CreateTablet for tablet 7d265495bde7475c8d06293d5881193a (DEFAULT_TABLE table=test-table [id=0cc5d9e394ba47b1a141d1948fae9a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:12.479206 27689 tablet_service.cc:1467] Processing CreateTablet for tablet 7d265495bde7475c8d06293d5881193a (DEFAULT_TABLE table=test-table [id=0cc5d9e394ba47b1a141d1948fae9a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:12.480479 27689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d265495bde7475c8d06293d5881193a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:12.480484 27428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d265495bde7475c8d06293d5881193a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:12.490994 27772 tablet_bootstrap.cc:492] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba: Bootstrap starting.
I20260927 13:56:12.494120 27772 tablet_bootstrap.cc:654] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:12.494482 27773 tablet_bootstrap.cc:492] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c: Bootstrap starting.
I20260927 13:56:12.495872 27772 log.cc:824] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:12.498018 27773 tablet_bootstrap.cc:654] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:12.498222 27774 tablet_bootstrap.cc:492] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25: Bootstrap starting.
I20260927 13:56:12.498787 27772 tablet_bootstrap.cc:492] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba: No bootstrap required, opened a new log
I20260927 13:56:12.499044 27772 ts_tablet_manager.cc:1397] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260927 13:56:12.499228 27773 log.cc:824] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:12.501389 27774 tablet_bootstrap.cc:654] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:12.502135 27773 tablet_bootstrap.cc:492] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c: No bootstrap required, opened a new log
I20260927 13:56:12.502409 27773 ts_tablet_manager.cc:1397] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 13:56:12.502545 27774 log.cc:824] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:12.505424 27774 tablet_bootstrap.cc:492] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25: No bootstrap required, opened a new log
I20260927 13:56:12.505896 27774 ts_tablet_manager.cc:1397] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 13:56:12.507157 27772 raft_consensus.cc:348] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d393d7841584de7a9252c063cb8db25" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45421 } } peers { permanent_uuid: "4877da551a0547398ee7bc7c746442ba" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 38011 } } peers { permanent_uuid: "15324440106e4e0fbd903bebfb37514c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 41023 } }
I20260927 13:56:12.507522 27772 raft_consensus.cc:374] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:12.507645 27772 raft_consensus.cc:729] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4877da551a0547398ee7bc7c746442ba, State: Initialized, Role: FOLLOWER
I20260927 13:56:12.508231 27772 consensus_queue.cc:260] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d393d7841584de7a9252c063cb8db25" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45421 } } peers { permanent_uuid: "4877da551a0547398ee7bc7c746442ba" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 38011 } } peers { permanent_uuid: "15324440106e4e0fbd903bebfb37514c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 41023 } }
I20260927 13:56:12.509297 27624 heartbeater.cc:503] Master 127.22.26.254:43281 was elected leader, sending a full tablet report...
I20260927 13:56:12.509842 27772 ts_tablet_manager.cc:1428] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 13:56:12.512463 27773 raft_consensus.cc:348] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d393d7841584de7a9252c063cb8db25" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45421 } } peers { permanent_uuid: "4877da551a0547398ee7bc7c746442ba" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 38011 } } peers { permanent_uuid: "15324440106e4e0fbd903bebfb37514c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 41023 } }
I20260927 13:56:12.512866 27773 raft_consensus.cc:374] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:12.512972 27773 raft_consensus.cc:729] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15324440106e4e0fbd903bebfb37514c, State: Initialized, Role: FOLLOWER
I20260927 13:56:12.513564 27773 consensus_queue.cc:260] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d393d7841584de7a9252c063cb8db25" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45421 } } peers { permanent_uuid: "4877da551a0547398ee7bc7c746442ba" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 38011 } } peers { permanent_uuid: "15324440106e4e0fbd903bebfb37514c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 41023 } }
I20260927 13:56:12.514618 27493 heartbeater.cc:503] Master 127.22.26.254:43281 was elected leader, sending a full tablet report...
I20260927 13:56:12.516371 27773 ts_tablet_manager.cc:1428] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 13:56:12.518262 27774 raft_consensus.cc:348] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d393d7841584de7a9252c063cb8db25" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45421 } } peers { permanent_uuid: "4877da551a0547398ee7bc7c746442ba" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 38011 } } peers { permanent_uuid: "15324440106e4e0fbd903bebfb37514c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 41023 } }
I20260927 13:56:12.518564 27774 raft_consensus.cc:374] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:12.518687 27774 raft_consensus.cc:729] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d393d7841584de7a9252c063cb8db25, State: Initialized, Role: FOLLOWER
I20260927 13:56:12.519366 27774 consensus_queue.cc:260] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d393d7841584de7a9252c063cb8db25" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45421 } } peers { permanent_uuid: "4877da551a0547398ee7bc7c746442ba" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 38011 } } peers { permanent_uuid: "15324440106e4e0fbd903bebfb37514c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 41023 } }
I20260927 13:56:12.520304 27754 heartbeater.cc:503] Master 127.22.26.254:43281 was elected leader, sending a full tablet report...
I20260927 13:56:12.520834 27774 ts_tablet_manager.cc:1428] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
W20260927 13:56:12.561990 27494 tablet.cc:2369] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:56:12.655018 27755 tablet.cc:2369] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:56:12.735039 27625 tablet.cc:2369] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:56:12.887660 27780 raft_consensus.cc:482] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:56:12.887980 27780 raft_consensus.cc:504] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d393d7841584de7a9252c063cb8db25" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45421 } } peers { permanent_uuid: "4877da551a0547398ee7bc7c746442ba" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 38011 } } peers { permanent_uuid: "15324440106e4e0fbd903bebfb37514c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 41023 } }
I20260927 13:56:12.889557 27780 leader_election.cc:288] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4877da551a0547398ee7bc7c746442ba (127.22.26.194:38011), 15324440106e4e0fbd903bebfb37514c (127.22.26.193:41023)
I20260927 13:56:12.892338 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.890030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:41023 (local address 127.22.26.195:38323)
0927 13:56:12.890203 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:12.890229 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:12.890340 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:12.890763 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:12.890768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:12.890798 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:12.891505 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:56:12.891507 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:12.891522 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:56:12.891565 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:12.891933 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:12.891940 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:12.891946 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 13:56:12.892046 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 13:56:12.892088 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 13:56:12.892375 27771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.890128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:38323 (local address 127.22.26.193:41023)
0927 13:56:12.890320 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:12.890325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:12.890341 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:56:12.890498 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:12.890504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:12.890576 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:12.890709 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:12.891728 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:12.891733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:12.891736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:12.891752 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:12.891783 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:12.891795 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:12.891949 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:12.892052 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 13:56:12.892111 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:56:12.893042 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.890064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:56661 (local address 127.22.26.194:38011)
0927 13:56:12.890256 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:12.890262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:12.890990 (+ 728us) server_negotiation.cc:407] Connection header received
0927 13:56:12.891149 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:12.891159 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:12.891207 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:12.891325 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:12.892267 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:12.892273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:12.892277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:12.892295 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:12.892335 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:12.892353 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:12.892479 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:12.892703 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 13:56:12.892765 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:56:12.893126 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.890057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:38011 (local address 127.22.26.195:56661)
0927 13:56:12.890882 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:12.890896 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:12.891011 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:12.891344 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:12.891349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:12.891373 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:12.892041 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:56:12.892044 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:12.892061 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:56:12.892105 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:12.892535 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:12.892539 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:12.892545 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 13:56:12.892697 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 13:56:12.892746 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":200,"thread_start_us":306,"threads_started":1}
I20260927 13:56:12.893435 27448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d265495bde7475c8d06293d5881193a" candidate_uuid: "6d393d7841584de7a9252c063cb8db25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15324440106e4e0fbd903bebfb37514c" is_pre_election: true
I20260927 13:56:12.894047 27448 raft_consensus.cc:2454] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d393d7841584de7a9252c063cb8db25 in term 0.
I20260927 13:56:12.894066 27579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d265495bde7475c8d06293d5881193a" candidate_uuid: "6d393d7841584de7a9252c063cb8db25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4877da551a0547398ee7bc7c746442ba" is_pre_election: true
I20260927 13:56:12.894593 27579 raft_consensus.cc:2454] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d393d7841584de7a9252c063cb8db25 in term 0.
I20260927 13:56:12.894889 27644 leader_election.cc:302] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15324440106e4e0fbd903bebfb37514c, 6d393d7841584de7a9252c063cb8db25; no voters:
I20260927 13:56:12.895355 27780 raft_consensus.cc:2781] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:56:12.895499 27780 raft_consensus.cc:482] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:56:12.895594 27780 raft_consensus.cc:3037] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:12.897698 27780 raft_consensus.cc:504] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d393d7841584de7a9252c063cb8db25" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45421 } } peers { permanent_uuid: "4877da551a0547398ee7bc7c746442ba" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 38011 } } peers { permanent_uuid: "15324440106e4e0fbd903bebfb37514c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 41023 } }
I20260927 13:56:12.898305 27780 leader_election.cc:288] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [CANDIDATE]: Term 1 election: Requested vote from peers 4877da551a0547398ee7bc7c746442ba (127.22.26.194:38011), 15324440106e4e0fbd903bebfb37514c (127.22.26.193:41023)
I20260927 13:56:12.898661 27579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d265495bde7475c8d06293d5881193a" candidate_uuid: "6d393d7841584de7a9252c063cb8db25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4877da551a0547398ee7bc7c746442ba"
I20260927 13:56:12.898809 27448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d265495bde7475c8d06293d5881193a" candidate_uuid: "6d393d7841584de7a9252c063cb8db25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15324440106e4e0fbd903bebfb37514c"
I20260927 13:56:12.898865 27579 raft_consensus.cc:3037] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:12.899011 27448 raft_consensus.cc:3037] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:12.902092 27579 raft_consensus.cc:2454] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d393d7841584de7a9252c063cb8db25 in term 1.
I20260927 13:56:12.902094 27448 raft_consensus.cc:2454] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d393d7841584de7a9252c063cb8db25 in term 1.
I20260927 13:56:12.902467 27644 leader_election.cc:302] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15324440106e4e0fbd903bebfb37514c, 6d393d7841584de7a9252c063cb8db25; no voters:
I20260927 13:56:12.902773 27780 raft_consensus.cc:2781] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:12.903223 27780 raft_consensus.cc:686] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [term 1 LEADER]: Becoming Leader. State: Replica: 6d393d7841584de7a9252c063cb8db25, State: Running, Role: LEADER
I20260927 13:56:12.903673 27780 consensus_queue.cc:237] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d393d7841584de7a9252c063cb8db25" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45421 } } peers { permanent_uuid: "4877da551a0547398ee7bc7c746442ba" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 38011 } } peers { permanent_uuid: "15324440106e4e0fbd903bebfb37514c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 41023 } }
I20260927 13:56:12.908200 27306 catalog_manager.cc:5495] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d393d7841584de7a9252c063cb8db25 (127.22.26.195). New cstate: current_term: 1 leader_uuid: "6d393d7841584de7a9252c063cb8db25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d393d7841584de7a9252c063cb8db25" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 45421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4877da551a0547398ee7bc7c746442ba" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15324440106e4e0fbd903bebfb37514c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 41023 } health_report { overall_health: UNKNOWN } } }
I20260927 13:56:12.928224 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.926076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.22.26.195:45421)
0927 13:56:12.926245 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:12.926250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:12.926370 (+ 120us) server_negotiation.cc:407] Connection header received
0927 13:56:12.926594 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:12.926598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:12.926677 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:12.926852 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:12.927729 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:12.927733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:12.927736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:12.927750 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:12.927782 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:12.927794 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:12.927932 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:12.927992 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 13:56:12.928037 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:56:12.936610 27579 raft_consensus.cc:1261] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba [term 1 FOLLOWER]: Refusing update from remote peer 6d393d7841584de7a9252c063cb8db25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:12.936633 27448 raft_consensus.cc:1261] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c [term 1 FOLLOWER]: Refusing update from remote peer 6d393d7841584de7a9252c063cb8db25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:12.937659 27780 consensus_queue.cc:1035] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15324440106e4e0fbd903bebfb37514c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 41023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:12.937994 27782 consensus_queue.cc:1035] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4877da551a0547398ee7bc7c746442ba" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:12.953469 27787 mvcc.cc:205] Tried to move back new op lower bound from 7333959159538511872 to 7333959159417344000. Current Snapshot: MvccSnapshot[applied={T|T < 7333959159538511872}]
I20260927 13:56:12.961584 27307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:12.961829 27307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:12.965047 27307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de7aebf51dbe4d82a2bfc2241e25e8ea: Sending DeleteTablet for 3 replicas of tablet 7d265495bde7475c8d06293d5881193a
I20260927 13:56:12.966234 27689 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d265495bde7475c8d06293d5881193a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:12 UTC) from {username='slave'} at 127.0.0.1:32860
I20260927 13:56:12.966274 27428 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d265495bde7475c8d06293d5881193a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:12 UTC) from {username='slave'} at 127.0.0.1:50340
I20260927 13:56:12.966526 27559 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d265495bde7475c8d06293d5881193a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:12 UTC) from {username='slave'} at 127.0.0.1:55868
I20260927 13:56:12.968863 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 27366
I20260927 13:56:12.997706 27771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:12.970466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.22.26.193:41023)
0927 13:56:12.970692 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:12.970699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:12.995027 (+ 24328us) server_negotiation.cc:407] Connection header received
0927 13:56:12.995545 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:12.995551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:12.995654 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:12.995725 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:12.996961 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:12.996967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:12.996972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:12.996992 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:12.997030 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:12.997044 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:12.997134 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:12.997355 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 13:56:12.997425 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:56:12.998930 27488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:13.009891 27795 tablet_replica.cc:330] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25: stopping tablet replica
I20260927 13:56:13.011822 27795 raft_consensus.cc:2229] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:56:13.013285 27795 raft_consensus.cc:2258] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:13.016901 27795 ts_tablet_manager.cc:1905] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:13.020864 27796 tablet_replica.cc:330] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba: stopping tablet replica
I20260927 13:56:13.022894 27796 raft_consensus.cc:2229] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:13.029453 27795 ts_tablet_manager.cc:1918] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:13.029987 27796 raft_consensus.cc:2258] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:13.032040 27796 ts_tablet_manager.cc:1905] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:13.035274 27795 log.cc:1196] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/7d265495bde7475c8d06293d5881193a
I20260927 13:56:13.035904 27795 ts_tablet_manager.cc:1939] T 7d265495bde7475c8d06293d5881193a P 6d393d7841584de7a9252c063cb8db25: Deleting consensus metadata
I20260927 13:56:13.037819 27796 ts_tablet_manager.cc:1918] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:13.037798 27291 catalog_manager.cc:4841] TS 6d393d7841584de7a9252c063cb8db25 (127.22.26.195:45421): tablet 7d265495bde7475c8d06293d5881193a (table test-table [id=0cc5d9e394ba47b1a141d1948fae9a16]) successfully deleted
I20260927 13:56:13.038044 27796 log.cc:1196] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/7d265495bde7475c8d06293d5881193a
I20260927 13:56:13.038656 27796 ts_tablet_manager.cc:1939] T 7d265495bde7475c8d06293d5881193a P 4877da551a0547398ee7bc7c746442ba: Deleting consensus metadata
I20260927 13:56:13.040328 27294 catalog_manager.cc:4841] TS 4877da551a0547398ee7bc7c746442ba (127.22.26.194:38011): tablet 7d265495bde7475c8d06293d5881193a (table test-table [id=0cc5d9e394ba47b1a141d1948fae9a16]) successfully deleted
I20260927 13:56:13.117690 27794 tablet_replica.cc:330] T 7d265495bde7475c8d06293d5881193a P 15324440106e4e0fbd903bebfb37514c: stopping tablet replica
I20260927 13:56:13.117760 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 27366
I20260927 13:56:13.140031 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 27497
W20260927 13:56:13.139998 27293 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.26.193:41023 (error 108)
I20260927 13:56:13.142323 27800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:13.141530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:41023 (local address 127.0.0.1:50368)
0927 13:56:13.141991 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:13.142094 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":251,"threads_started":1}
W20260927 13:56:13.142606 27293 catalog_manager.cc:4584] TS 15324440106e4e0fbd903bebfb37514c (127.22.26.193:41023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d265495bde7475c8d06293d5881193a: Network error: Client connection negotiation failed: client connection to 127.22.26.193:41023: connect: Connection refused (error 111)
I20260927 13:56:13.147722 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:13.141221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.22.26.194:38011)
0927 13:56:13.141416 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:13.141422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:13.141442 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:56:13.141507 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:13.141513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:13.141620 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:13.141724 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:13.147004 (+ 5280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:13.147009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:13.147012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:13.147027 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:13.147059 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:13.147072 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:13.147237 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:13.147358 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 13:56:13.147408 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:56:13.148343 27619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:13.196430 27800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:13.195869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:41023 (local address 127.0.0.1:50370)
0927 13:56:13.196110 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:13.196251 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 13:56:13.197669 27800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:13.197238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:41023 (local address 127.0.0.1:50380)
0927 13:56:13.197417 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:13.197506 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 13:56:13.236765 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 27497
I20260927 13:56:13.254110 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 27627
I20260927 13:56:13.257334 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:13.254929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.22.26.195:45421)
0927 13:56:13.255405 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:13.255410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:13.255424 (+ 14us) server_negotiation.cc:407] Connection header received
0927 13:56:13.255500 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:13.255504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:13.255547 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:13.255658 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:13.256650 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:13.256654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:13.256657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:13.256672 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:13.256710 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:13.256721 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:13.256886 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:13.256988 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 13:56:13.257046 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377}
I20260927 13:56:13.258001 27749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:13.258077 27800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:13.257571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:41023 (local address 127.0.0.1:50390)
0927 13:56:13.257835 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:13.257927 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260927 13:56:13.259298 27800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:13.258820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:41023 (local address 127.0.0.1:50398)
0927 13:56:13.259020 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:13.259146 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 13:56:13.339946 27800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:13.339493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:41023 (local address 127.0.0.1:50412)
0927 13:56:13.339712 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:13.339807 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 13:56:13.341018 27800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:13.340636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:41023 (local address 127.0.0.1:50424)
0927 13:56:13.340822 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:13.340889 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 13:56:13.380229 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 27627
I20260927 13:56:13.395794 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 27276
I20260927 13:56:13.397106 27337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:13.507424 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 27276
2026-09-27T13:56:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2977 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 13:56:13.549724 22635 mini_oidc.cc:152] Starting JWKS server
I20260927 13:56:13.553308 22635 webserver.cc:460] Webserver started at https://127.0.0.1:44649/ using document root <none> and password file <none>
I20260927 13:56:13.553658 22635 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 13:56:13.554342 22635 webserver.cc:460] Webserver started at http://127.0.0.1:37089/ using document root <none> and password file <none>
2026-09-27T13:56:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:56:13Z Disabled control of system clock
I20260927 13:56:13.580928 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:33931
--webserver_interface=127.22.26.254
--webserver_port=0
--builtin_ntp_servers=127.22.26.212:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:33931
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44649/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/testchainca.cert with env {}
W20260927 13:56:13.898272 27809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:13.898554 27809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:13.898639 27809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:13.905787 27809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:13.905902 27809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:13.905939 27809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:13.905988 27809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:13.906032 27809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 13:56:13.906096 27809 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 13:56:13.906136 27809 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44649/jwks.json
I20260927 13:56:13.916110 27809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:33931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:33931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44649/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:13.917435 27809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:13.918705 27809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:13.930706 27817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:13.930749 27815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:13.930825 27814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:13.931474 27809 server_base.cc:1034] running on GCE node
I20260927 13:56:13.932368 27809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:13.933845 27809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:13.935130 27809 hybrid_clock.cc:648] HybridClock initialized: now 1790517373935037 us; error 44 us; skew 500 ppm
I20260927 13:56:13.935490 27809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:56:13.937840 27809 webserver.cc:460] Webserver started at http://127.22.26.254:36847/ using document root <none> and password file <none>
I20260927 13:56:13.938550 27809 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:13.938692 27809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:13.939003 27809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:13.941928 27809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "f7d4582c90b5462bb6a228d86bbb7745"
format_stamp: "Formatted at 2026-09-27 13:56:13 on dist-test-slave-gkks"
server_key: "032ebd9659d80140021a7fc98b33b09b"
server_key_iv: "e3ae5dfff7ead3aa44f594a05d49ba74"
server_key_version: "encryptionkey@0"
I20260927 13:56:13.942776 27809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "f7d4582c90b5462bb6a228d86bbb7745"
format_stamp: "Formatted at 2026-09-27 13:56:13 on dist-test-slave-gkks"
server_key: "032ebd9659d80140021a7fc98b33b09b"
server_key_iv: "e3ae5dfff7ead3aa44f594a05d49ba74"
server_key_version: "encryptionkey@0"
I20260927 13:56:13.948240 27809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 13:56:13.951779 27823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:13.953101 27809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:56:13.953292 27809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "f7d4582c90b5462bb6a228d86bbb7745"
format_stamp: "Formatted at 2026-09-27 13:56:13 on dist-test-slave-gkks"
server_key: "032ebd9659d80140021a7fc98b33b09b"
server_key_iv: "e3ae5dfff7ead3aa44f594a05d49ba74"
server_key_version: "encryptionkey@0"
I20260927 13:56:13.953510 27809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:13.976128 27809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:13.976922 27809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:13.977221 27809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:13.995507 27809 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:33931
I20260927 13:56:13.995540 27874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:33931 every 8 connection(s)
I20260927 13:56:13.996829 27809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:56:13.998283 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 27809
I20260927 13:56:13.998476 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:56:13.999311 22635 external_mini_cluster.cc:1428] Setting key 290497bc73f22b6a283055e3a1199ab1
I20260927 13:56:14.001554 27875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:14.013022 27878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:14.001189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43348 (local address 127.22.26.254:33931)
0927 13:56:14.002191 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:14.002202 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:14.002242 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:56:14.003875 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:14.003895 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:14.004332 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:14.004664 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:14.005304 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:14.006617 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:14.008543 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:14.009175 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:14.009323 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:14.010641 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:14.010648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:14.010660 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:14.011271 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:14.011351 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:14.011379 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:14.011467 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:14.011718 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 13:56:14.012058 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":415,"threads_started":1}
I20260927 13:56:14.014190 27875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745: Bootstrap starting.
I20260927 13:56:14.016954 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:56:14.017750 27875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:14.018733 27875 log.cc:824] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:14.021690 27875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745: No bootstrap required, opened a new log
I20260927 13:56:14.029028 27875 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d4582c90b5462bb6a228d86bbb7745" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33931 } }
I20260927 13:56:14.029461 27875 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:14.029572 27875 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d4582c90b5462bb6a228d86bbb7745, State: Initialized, Role: FOLLOWER
I20260927 13:56:14.030200 27875 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d4582c90b5462bb6a228d86bbb7745" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33931 } }
I20260927 13:56:14.030438 27875 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:56:14.030532 27875 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:56:14.030658 27875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:14.032894 27875 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d4582c90b5462bb6a228d86bbb7745" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33931 } }
I20260927 13:56:14.033367 27875 leader_election.cc:302] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7d4582c90b5462bb6a228d86bbb7745; no voters:
I20260927 13:56:14.033905 27875 leader_election.cc:288] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:56:14.034075 27880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:14.034775 27880 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [term 1 LEADER]: Becoming Leader. State: Replica: f7d4582c90b5462bb6a228d86bbb7745, State: Running, Role: LEADER
I20260927 13:56:14.035423 27880 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d4582c90b5462bb6a228d86bbb7745" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33931 } }
I20260927 13:56:14.036068 27875 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:14.038396 27882 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7d4582c90b5462bb6a228d86bbb7745. Latest consensus state: current_term: 1 leader_uuid: "f7d4582c90b5462bb6a228d86bbb7745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d4582c90b5462bb6a228d86bbb7745" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33931 } } }
I20260927 13:56:14.038422 27881 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7d4582c90b5462bb6a228d86bbb7745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d4582c90b5462bb6a228d86bbb7745" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33931 } } }
I20260927 13:56:14.038947 27881 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:14.038954 27882 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:14.044431 27886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:14.049021 27886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:14.057844 27886 catalog_manager.cc:1347] Generated new cluster ID: 182d5df30c9f421690cb05970662098f
I20260927 13:56:14.057986 27886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:14.082933 27886 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:56:14.084981 27886 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:14.091879 27886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7d4582c90b5462bb6a228d86bbb7745: Generated new TSK 0
I20260927 13:56:14.092729 27886 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 13:56:14.139586 27905 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 13:56:14.139762 27905 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 13:56:14.141852 27878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:14.117936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.22.26.254:33931)
0927 13:56:14.118137 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:14.118143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:14.118573 (+ 430us) server_negotiation.cc:407] Connection header received
0927 13:56:14.118858 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:14.118864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:14.118963 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:14.119179 (+ 216us) server_negotiation.cc:226] Negotiated authn=JWT
0927 13:56:14.119684 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:14.120847 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:14.122242 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:14.122967 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:14.123120 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:14.123706 (+ 586us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 13:56:14.141149 (+ 17443us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 13:56:14.141517 (+ 368us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 13:56:14.141981 27878 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 13:56:14.143410 27904 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:14.117802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:33931 (local address 127.0.0.1:43356)
0927 13:56:14.118470 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:14.118486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:14.118628 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:14.119205 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:14.119209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:14.119220 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 13:56:14.119539 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:14.119548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:14.120989 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:14.120993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:14.122100 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:14.122111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:14.123201 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:14.123206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:14.123341 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:14.123398 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 13:56:14.141459 (+ 18061us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 13:56:14.141761 (+ 302us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.26.254:33931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":320,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.22.26.254:33931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 13:56:14.145265 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 27809
I20260927 13:56:14.146468 27870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:14.241484 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 27809
2026-09-27T13:56:14Z chronyd exiting
I20260927 13:56:14.292379 22635 test_util.cc:175] -----------------------------------------------
I20260927 13:56:14.292541 22635 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790517330258165-22635-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (748 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 13:56:14.296388 22635 mini_oidc.cc:152] Starting JWKS server
I20260927 13:56:14.298273 22635 webserver.cc:460] Webserver started at https://127.0.0.1:35881/ using document root <none> and password file <none>
I20260927 13:56:14.298533 22635 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 13:56:14.299286 22635 webserver.cc:460] Webserver started at http://127.0.0.1:35279/ using document root <none> and password file <none>
2026-09-27T13:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:56:14Z Disabled control of system clock
I20260927 13:56:14.326727 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:44443
--webserver_interface=127.22.26.254
--webserver_port=0
--builtin_ntp_servers=127.22.26.212:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:44443
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35881/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/testchainca.cert with env {}
W20260927 13:56:14.642477 27912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:14.642812 27912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:14.642983 27912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:14.650306 27912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:14.650516 27912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:14.650619 27912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:14.650733 27912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:14.650820 27912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 13:56:14.650907 27912 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 13:56:14.650965 27912 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35881/jwks.json
I20260927 13:56:14.660815 27912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:44443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:44443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35881/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:14.662102 27912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:14.663424 27912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:14.674993 27920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:14.675136 27917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:14.675119 27918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:14.676117 27912 server_base.cc:1034] running on GCE node
I20260927 13:56:14.676908 27912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:14.678303 27912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:14.679557 27912 hybrid_clock.cc:648] HybridClock initialized: now 1790517374679530 us; error 40 us; skew 500 ppm
I20260927 13:56:14.679994 27912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:56:14.682267 27912 webserver.cc:460] Webserver started at http://127.22.26.254:40621/ using document root <none> and password file <none>
I20260927 13:56:14.682955 27912 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:14.683120 27912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:14.683467 27912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:14.686270 27912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "e6c1d644860245e6949eca0accb376b4"
format_stamp: "Formatted at 2026-09-27 13:56:14 on dist-test-slave-gkks"
server_key: "2dd1873540c158ca0e2e1ffa0ac6b15a"
server_key_iv: "15be97726b68727cec39d1e50a7cfce9"
server_key_version: "encryptionkey@0"
I20260927 13:56:14.687047 27912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "e6c1d644860245e6949eca0accb376b4"
format_stamp: "Formatted at 2026-09-27 13:56:14 on dist-test-slave-gkks"
server_key: "2dd1873540c158ca0e2e1ffa0ac6b15a"
server_key_iv: "15be97726b68727cec39d1e50a7cfce9"
server_key_version: "encryptionkey@0"
I20260927 13:56:14.692153 27912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 13:56:14.695549 27926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:14.696972 27912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 13:56:14.697149 27912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "e6c1d644860245e6949eca0accb376b4"
format_stamp: "Formatted at 2026-09-27 13:56:14 on dist-test-slave-gkks"
server_key: "2dd1873540c158ca0e2e1ffa0ac6b15a"
server_key_iv: "15be97726b68727cec39d1e50a7cfce9"
server_key_version: "encryptionkey@0"
I20260927 13:56:14.697366 27912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:14.723652 27912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:14.724474 27912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:14.724793 27912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:14.743249 27912 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:44443
I20260927 13:56:14.743294 27977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:44443 every 8 connection(s)
I20260927 13:56:14.744570 27912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:56:14.748405 27978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:14.754804 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 27912
I20260927 13:56:14.755038 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:56:14.755730 22635 external_mini_cluster.cc:1428] Setting key 07fbad1f6aeb72e0240435d020ec9b70
I20260927 13:56:14.759327 27978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4: Bootstrap starting.
I20260927 13:56:14.762645 27978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:14.763972 27978 log.cc:824] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:14.766979 27978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4: No bootstrap required, opened a new log
I20260927 13:56:14.769623 27981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:14.757794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.22.26.254:44443)
0927 13:56:14.758754 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:14.758764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:14.758806 (+ 42us) server_negotiation.cc:407] Connection header received
0927 13:56:14.760521 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:14.760543 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:14.761008 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:14.761326 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:14.761915 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:14.763335 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:14.765117 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:14.765883 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:14.766086 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:14.767380 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:14.767389 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:14.767404 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:14.767918 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:14.767988 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:14.768008 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:14.768152 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:14.768432 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 13:56:14.768705 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":352,"threads_started":1}
I20260927 13:56:14.773483 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:56:14.774869 27978 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c1d644860245e6949eca0accb376b4" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44443 } }
I20260927 13:56:14.775285 27978 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:14.775385 27978 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c1d644860245e6949eca0accb376b4, State: Initialized, Role: FOLLOWER
I20260927 13:56:14.775985 27978 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c1d644860245e6949eca0accb376b4" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44443 } }
I20260927 13:56:14.776245 27978 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:56:14.776402 27978 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:56:14.776597 27978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:14.778910 27978 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c1d644860245e6949eca0accb376b4" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44443 } }
I20260927 13:56:14.779421 27978 leader_election.cc:302] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6c1d644860245e6949eca0accb376b4; no voters:
I20260927 13:56:14.780007 27978 leader_election.cc:288] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:56:14.780121 27983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:14.781551 27983 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [term 1 LEADER]: Becoming Leader. State: Replica: e6c1d644860245e6949eca0accb376b4, State: Running, Role: LEADER
I20260927 13:56:14.782051 27978 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:14.782033 27983 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c1d644860245e6949eca0accb376b4" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44443 } }
I20260927 13:56:14.785010 27985 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6c1d644860245e6949eca0accb376b4. Latest consensus state: current_term: 1 leader_uuid: "e6c1d644860245e6949eca0accb376b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c1d644860245e6949eca0accb376b4" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44443 } } }
I20260927 13:56:14.785420 27985 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:14.785944 27984 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6c1d644860245e6949eca0accb376b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c1d644860245e6949eca0accb376b4" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44443 } } }
I20260927 13:56:14.786238 27984 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:14.786356 27992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:14.792922 27992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:14.802781 27992 catalog_manager.cc:1347] Generated new cluster ID: b1bb1cd8f61e4cf1a1276fb0812bf1cf
I20260927 13:56:14.802896 27992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:14.815994 27992 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:56:14.817584 27992 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:14.822783 27992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6c1d644860245e6949eca0accb376b4: Generated new TSK 0
I20260927 13:56:14.823570 27992 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 13:56:14.854683 28008 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 13:56:14.854817 28008 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 13:56:14.856026 27981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:14.836022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53322 (local address 127.22.26.254:44443)
0927 13:56:14.836244 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:14.836251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:14.836610 (+ 359us) server_negotiation.cc:407] Connection header received
0927 13:56:14.836871 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:14.836875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:14.836953 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:14.837102 (+ 149us) server_negotiation.cc:226] Negotiated authn=JWT
0927 13:56:14.837576 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:14.838287 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:14.839918 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:14.840400 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:14.840581 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:14.841108 (+ 527us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 13:56:14.855343 (+ 14235us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 13:56:14.855742 (+ 399us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53322: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 13:56:14.856153 27981 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53322: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 13:56:14.856226 28007 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:14.835947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44443 (local address 127.0.0.1:53322)
0927 13:56:14.836506 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:14.836526 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:14.836689 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:14.837118 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:14.837123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:14.837134 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 13:56:14.837411 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:14.837420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:14.838435 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:14.838440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:14.839759 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:14.839773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:14.840634 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:14.840640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:14.840816 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:14.840852 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 13:56:14.855569 (+ 14717us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 13:56:14.855834 (+ 265us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.26.254:44443: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":335,"threads_started":1}
I20260927 13:56:14.857635 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 27912
I20260927 13:56:14.858783 27973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:14.950538 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 27912
2026-09-27T13:56:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (709 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 13:56:15.005663 22635 mini_oidc.cc:152] Starting JWKS server
I20260927 13:56:15.007623 22635 webserver.cc:460] Webserver started at https://127.0.0.1:36581/ using document root <none> and password file <none>
I20260927 13:56:15.007866 22635 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 13:56:15.008489 22635 webserver.cc:460] Webserver started at http://127.0.0.1:37925/ using document root <none> and password file <none>
2026-09-27T13:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:56:15Z Disabled control of system clock
I20260927 13:56:15.035004 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42731
--webserver_interface=127.22.26.254
--webserver_port=0
--builtin_ntp_servers=127.22.26.212:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:42731
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36581/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 13:56:15.357555 28015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:15.357847 28015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:15.357934 28015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:15.365073 28015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:15.365178 28015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:15.365216 28015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:15.365283 28015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:15.365320 28015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 13:56:15.365379 28015 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 13:56:15.365418 28015 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36581/jwks.json
I20260927 13:56:15.374459 28015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:42731
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36581/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:15.375752 28015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:15.377445 28015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:15.388970 28021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:15.389087 28020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:15.389151 28023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:15.390375 28015 server_base.cc:1034] running on GCE node
I20260927 13:56:15.391290 28015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:15.392654 28015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:15.393853 28015 hybrid_clock.cc:648] HybridClock initialized: now 1790517375393793 us; error 50 us; skew 500 ppm
I20260927 13:56:15.394194 28015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:56:15.396548 28015 webserver.cc:460] Webserver started at http://127.22.26.254:43799/ using document root <none> and password file <none>
I20260927 13:56:15.397229 28015 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:15.397341 28015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:15.397682 28015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:15.400590 28015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "a7079849b1024398aeda76fe889f0475"
format_stamp: "Formatted at 2026-09-27 13:56:15 on dist-test-slave-gkks"
server_key: "6afe68cec8b110220fa4a40b529ee427"
server_key_iv: "3596ea5a3a2d7d62c6fb0fd536b07976"
server_key_version: "encryptionkey@0"
I20260927 13:56:15.401449 28015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "a7079849b1024398aeda76fe889f0475"
format_stamp: "Formatted at 2026-09-27 13:56:15 on dist-test-slave-gkks"
server_key: "6afe68cec8b110220fa4a40b529ee427"
server_key_iv: "3596ea5a3a2d7d62c6fb0fd536b07976"
server_key_version: "encryptionkey@0"
I20260927 13:56:15.406442 28015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:56:15.409914 28029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:15.411401 28015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:15.411586 28015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "a7079849b1024398aeda76fe889f0475"
format_stamp: "Formatted at 2026-09-27 13:56:15 on dist-test-slave-gkks"
server_key: "6afe68cec8b110220fa4a40b529ee427"
server_key_iv: "3596ea5a3a2d7d62c6fb0fd536b07976"
server_key_version: "encryptionkey@0"
I20260927 13:56:15.411839 28015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:15.456383 28015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:15.457176 28015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:15.457434 28015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:15.476104 28015 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:42731
I20260927 13:56:15.476122 28080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:42731 every 8 connection(s)
I20260927 13:56:15.477387 28015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:56:15.480980 28081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:15.482360 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 28015
I20260927 13:56:15.482534 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:56:15.483247 22635 external_mini_cluster.cc:1428] Setting key 40d442e4e29b3a08258e8e2178b4ce0d
I20260927 13:56:15.494984 28081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475: Bootstrap starting.
I20260927 13:56:15.497742 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:15.485124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.22.26.254:42731)
0927 13:56:15.486165 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:15.486184 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:15.486251 (+ 67us) server_negotiation.cc:407] Connection header received
0927 13:56:15.487945 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:15.487977 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:15.488593 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:15.489013 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:15.489490 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:15.491415 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:15.492743 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:15.493720 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:15.493918 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:15.494952 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:15.494968 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:15.494985 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:15.495705 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:15.495819 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:15.495846 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:15.495956 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:15.496175 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 13:56:15.496476 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":316,"threads_started":1}
I20260927 13:56:15.498715 28081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:15.499794 28081 log.cc:824] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:15.501487 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:56:15.502892 28081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475: No bootstrap required, opened a new log
I20260927 13:56:15.510079 28081 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7079849b1024398aeda76fe889f0475" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42731 } }
I20260927 13:56:15.510375 28081 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:15.510443 28081 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7079849b1024398aeda76fe889f0475, State: Initialized, Role: FOLLOWER
I20260927 13:56:15.510970 28081 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7079849b1024398aeda76fe889f0475" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42731 } }
I20260927 13:56:15.511246 28081 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:56:15.511404 28081 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:56:15.511595 28081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:15.513608 28081 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7079849b1024398aeda76fe889f0475" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42731 } }
I20260927 13:56:15.514076 28081 leader_election.cc:302] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7079849b1024398aeda76fe889f0475; no voters:
I20260927 13:56:15.514667 28081 leader_election.cc:288] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:56:15.514844 28086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:15.515606 28086 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [term 1 LEADER]: Becoming Leader. State: Replica: a7079849b1024398aeda76fe889f0475, State: Running, Role: LEADER
I20260927 13:56:15.516232 28086 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7079849b1024398aeda76fe889f0475" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42731 } }
I20260927 13:56:15.516793 28081 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:15.518723 28088 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7079849b1024398aeda76fe889f0475. Latest consensus state: current_term: 1 leader_uuid: "a7079849b1024398aeda76fe889f0475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7079849b1024398aeda76fe889f0475" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42731 } } }
I20260927 13:56:15.519048 28088 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:15.519344 28087 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7079849b1024398aeda76fe889f0475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7079849b1024398aeda76fe889f0475" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42731 } } }
I20260927 13:56:15.519757 28087 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:15.522610 28091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:15.527904 28091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:15.535825 28091 catalog_manager.cc:1347] Generated new cluster ID: 2b0ce4adebe041fc98fa3e70475c701e
I20260927 13:56:15.535933 28091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:15.548776 28091 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:56:15.549949 28091 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:15.561662 28091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7079849b1024398aeda76fe889f0475: Generated new TSK 0
I20260927 13:56:15.562505 28091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:56:15.588733 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:15.580273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.22.26.254:42731)
0927 13:56:15.580458 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:15.580465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:15.580865 (+ 400us) server_negotiation.cc:407] Connection header received
0927 13:56:15.581072 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:15.581077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:15.581173 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:15.581340 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0927 13:56:15.581878 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:15.583038 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:15.584480 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:15.585205 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:15.585334 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:15.585790 (+ 456us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 13:56:15.588078 (+ 2288us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 13:56:15.588432 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51974: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 13:56:15.588847 28084 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51974: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 13:56:15.590488 28110 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:15.580111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42731 (local address 127.0.0.1:51974)
0927 13:56:15.580766 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:15.580782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:15.580896 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:15.581352 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:15.581356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:15.581365 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 13:56:15.581716 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:15.581726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:15.583203 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:15.583208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:15.584332 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:15.584344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:15.585385 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:15.585390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:15.585513 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:15.585543 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 13:56:15.588320 (+ 2777us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 13:56:15.590087 (+ 1767us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.26.254:42731: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":1489,"thread_start_us":398,"threads_started":1}
I20260927 13:56:15.591905 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 28015
I20260927 13:56:15.593017 28076 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 13:56:15.600498 28111 webserver.cc:533] Webserver: sslize failed:
E20260927 13:56:15.600579 28111 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 13:56:15.683835 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 28015
2026-09-27T13:56:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (733 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T13:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:56:15Z Disabled control of system clock
I20260927 13:56:15.765421 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:46373
--webserver_interface=127.22.26.254
--webserver_port=0
--builtin_ntp_servers=127.22.26.212:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:46373
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/testchainca.cert with env {}
W20260927 13:56:16.080192 28116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:16.080525 28116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:16.080690 28116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:16.087955 28116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:16.088058 28116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:16.088152 28116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:16.088239 28116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:16.088296 28116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 13:56:16.088426 28116 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 13:56:16.088508 28116 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 13:56:16.098133 28116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:46373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:46373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:16.099479 28116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:16.100801 28116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:16.112344 28121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:16.112286 28122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:16.112308 28124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:16.113148 28116 server_base.cc:1034] running on GCE node
I20260927 13:56:16.113878 28116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:16.115224 28116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:16.116420 28116 hybrid_clock.cc:648] HybridClock initialized: now 1790517376116374 us; error 46 us; skew 500 ppm
I20260927 13:56:16.116787 28116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:56:16.118906 28116 webserver.cc:460] Webserver started at http://127.22.26.254:37495/ using document root <none> and password file <none>
I20260927 13:56:16.119607 28116 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:16.119763 28116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:16.120057 28116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:16.122860 28116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "7d8d45c66c164103ad90d23dbc01bc79"
format_stamp: "Formatted at 2026-09-27 13:56:16 on dist-test-slave-gkks"
server_key: "0f6272a6871d23a34562990839919343"
server_key_iv: "cad2d2ce260bd3c9899954dde14528bb"
server_key_version: "encryptionkey@0"
I20260927 13:56:16.123708 28116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "7d8d45c66c164103ad90d23dbc01bc79"
format_stamp: "Formatted at 2026-09-27 13:56:16 on dist-test-slave-gkks"
server_key: "0f6272a6871d23a34562990839919343"
server_key_iv: "cad2d2ce260bd3c9899954dde14528bb"
server_key_version: "encryptionkey@0"
I20260927 13:56:16.128463 28116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:56:16.131916 28130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:16.133260 28116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:56:16.133447 28116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "7d8d45c66c164103ad90d23dbc01bc79"
format_stamp: "Formatted at 2026-09-27 13:56:16 on dist-test-slave-gkks"
server_key: "0f6272a6871d23a34562990839919343"
server_key_iv: "cad2d2ce260bd3c9899954dde14528bb"
server_key_version: "encryptionkey@0"
I20260927 13:56:16.133662 28116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:16.152113 28116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:16.152894 28116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:16.153170 28116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:16.171289 28116 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:46373
I20260927 13:56:16.171314 28181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:46373 every 8 connection(s)
I20260927 13:56:16.172652 28116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:56:16.176281 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:16.182618 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 28116
I20260927 13:56:16.182845 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:56:16.183568 22635 external_mini_cluster.cc:1428] Setting key 2548588cad3709896f48b32213bbb969
I20260927 13:56:16.186599 28182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79: Bootstrap starting.
I20260927 13:56:16.189947 28182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:16.191020 28182 log.cc:824] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:16.194166 28182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79: No bootstrap required, opened a new log
I20260927 13:56:16.196879 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:16.185417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.22.26.254:46373)
0927 13:56:16.186385 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:16.186401 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:16.186441 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:56:16.188162 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:16.188183 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:16.188640 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:16.188944 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:16.189548 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:16.190866 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:16.192501 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:16.193117 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:16.193249 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:16.194596 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:16.194603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:16.194619 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:56:16.195184 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:16.195284 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:56:16.195305 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:16.195390 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:16.195614 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 13:56:16.195949 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":351,"threads_started":1}
I20260927 13:56:16.200510 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:56:16.202133 28182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8d45c66c164103ad90d23dbc01bc79" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 46373 } }
I20260927 13:56:16.202497 28182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:16.202620 28182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d8d45c66c164103ad90d23dbc01bc79, State: Initialized, Role: FOLLOWER
I20260927 13:56:16.203303 28182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8d45c66c164103ad90d23dbc01bc79" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 46373 } }
I20260927 13:56:16.203562 28182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:56:16.203717 28182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:56:16.203899 28182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:16.205986 28182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8d45c66c164103ad90d23dbc01bc79" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 46373 } }
I20260927 13:56:16.206476 28182 leader_election.cc:302] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d8d45c66c164103ad90d23dbc01bc79; no voters:
I20260927 13:56:16.207041 28182 leader_election.cc:288] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:56:16.207253 28187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:16.208467 28187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [term 1 LEADER]: Becoming Leader. State: Replica: 7d8d45c66c164103ad90d23dbc01bc79, State: Running, Role: LEADER
I20260927 13:56:16.208978 28187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8d45c66c164103ad90d23dbc01bc79" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 46373 } }
I20260927 13:56:16.209298 28182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:16.212139 28189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d8d45c66c164103ad90d23dbc01bc79. Latest consensus state: current_term: 1 leader_uuid: "7d8d45c66c164103ad90d23dbc01bc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8d45c66c164103ad90d23dbc01bc79" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 46373 } } }
I20260927 13:56:16.212610 28189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:16.212265 28188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d8d45c66c164103ad90d23dbc01bc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8d45c66c164103ad90d23dbc01bc79" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 46373 } } }
I20260927 13:56:16.213214 28188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:16.215902 28196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:16.220494 28196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:16.229002 28196 catalog_manager.cc:1347] Generated new cluster ID: 77e53bd1edfd448599a87038f7cca33a
I20260927 13:56:16.229112 28196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:16.254786 28196 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:56:16.256677 28196 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:16.267524 28196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d8d45c66c164103ad90d23dbc01bc79: Generated new TSK 0
I20260927 13:56:16.268563 28196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:56:16.344193 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:16.286280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.22.26.254:46373)
0927 13:56:16.286458 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:16.286464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:16.286891 (+ 427us) server_negotiation.cc:407] Connection header received
0927 13:56:16.287073 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:16.287099 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:16.287189 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:16.287352 (+ 163us) server_negotiation.cc:226] Negotiated authn=JWT
0927 13:56:16.287863 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:16.289005 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:16.290417 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:16.291122 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:16.291239 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:16.291744 (+ 505us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 13:56:16.343414 (+ 51670us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 13:56:16.343847 (+ 433us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33582: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 13:56:16.344355 28185 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33582: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 13:56:16.344426 28211 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:16.286102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:46373 (local address 127.0.0.1:33582)
0927 13:56:16.286793 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:16.286808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:16.286923 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:16.287395 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:16.287399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:16.287411 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0927 13:56:16.287689 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:16.287700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:16.289162 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:16.289166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:16.290269 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:16.290282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:16.291320 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:16.291325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:16.291465 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:16.291499 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 13:56:16.343714 (+ 52215us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 13:56:16.344014 (+ 300us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.26.254:46373: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":397,"threads_started":1}
I20260927 13:56:16.345872 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 28116
I20260927 13:56:16.347219 28177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:16.445674 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 28116
2026-09-27T13:56:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (741 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (399 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1568 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:56:18 dist-test-slave-gkks krb5kdc[28227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:56:18 dist-test-slave-gkks krb5kdc[28227](info): set up 2 sockets
Sep 27 13:56:18 dist-test-slave-gkks krb5kdc[28227](info): commencing operation
krb5kdc: starting...
W20260927 13:56:19.504019 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:56:19 dist-test-slave-gkks krb5kdc[28227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:56:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:56:19.702966 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:44509
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:44509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:20.017395 28240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:20.017764 28240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:20.017928 28240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:20.025213 28240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:20.025350 28240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:20.025457 28240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:20.025571 28240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:20.025655 28240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:56:20.035334 28240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:44509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:44509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:20.036675 28240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:20.038005 28240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:20.049667 28245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:20.049750 28248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:20.049703 28246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:20.050679 28240 server_base.cc:1034] running on GCE node
I20260927 13:56:20.051478 28240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:20.052779 28240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:20.054057 28240 hybrid_clock.cc:648] HybridClock initialized: now 1790517380054024 us; error 42 us; skew 500 ppm
I20260927 13:56:20.054406 28240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:20 dist-test-slave-gkks krb5kdc[28227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517380, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:20.057456 28240 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:56:20.058811 28240 webserver.cc:460] Webserver started at http://127.22.26.254:34593/ using document root <none> and password file <none>
I20260927 13:56:20.059732 28240 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:20.059908 28240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:20.060256 28240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:20.063138 28240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "d4bbf161413f45bb8a7faf5432cc6622"
format_stamp: "Formatted at 2026-09-27 13:56:20 on dist-test-slave-gkks"
server_key: "f78e4473f4500b7f64611d29dca9a3bd"
server_key_iv: "4e63f7161f38ebf15724a4aca3422395"
server_key_version: "encryptionkey@0"
I20260927 13:56:20.063894 28240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "d4bbf161413f45bb8a7faf5432cc6622"
format_stamp: "Formatted at 2026-09-27 13:56:20 on dist-test-slave-gkks"
server_key: "f78e4473f4500b7f64611d29dca9a3bd"
server_key_iv: "4e63f7161f38ebf15724a4aca3422395"
server_key_version: "encryptionkey@0"
I20260927 13:56:20.068778 28240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 13:56:20.072062 28255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:20.073357 28240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:56:20.073529 28240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "d4bbf161413f45bb8a7faf5432cc6622"
format_stamp: "Formatted at 2026-09-27 13:56:20 on dist-test-slave-gkks"
server_key: "f78e4473f4500b7f64611d29dca9a3bd"
server_key_iv: "4e63f7161f38ebf15724a4aca3422395"
server_key_version: "encryptionkey@0"
I20260927 13:56:20.073714 28240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:20.091190 28240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:20.095541 28240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:20.095844 28240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:20.114504 28240 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:44509
I20260927 13:56:20.114527 28306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:44509 every 8 connection(s)
I20260927 13:56:20.115875 28240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:56:20.119674 28307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:20.121161 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 28240
I20260927 13:56:20.121418 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:56:20.122049 22635 external_mini_cluster.cc:1428] Setting key dda46e59de7a21554e4b3703f6838997
I20260927 13:56:20.131354 28307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622: Bootstrap starting.
I20260927 13:56:20.135113 28307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:20.136361 28307 log.cc:824] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622: Log is configured to *not* fsync() on all Append() calls
Sep 27 13:56:20 dist-test-slave-gkks krb5kdc[28227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:20.139338 28307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622: No bootstrap required, opened a new log
I20260927 13:56:20.145156 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:20.124024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.22.26.254:44509)
0927 13:56:20.124796 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:20.124807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:20.124851 (+ 44us) server_negotiation.cc:407] Connection header received
0927 13:56:20.126508 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:20.126529 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:20.126909 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:20.127273 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:20.129061 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:20.130357 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:20.132134 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:20.132805 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:20.132947 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:20.137952 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:20.137960 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:20.137974 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:20.138008 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:20.141154 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:20.141827 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:20.141836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:20.141841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:20.141998 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:20.142446 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:20.142453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:20.142460 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:20.142978 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:20.143326 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:20.143763 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 13:56:20.144153 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":357,"threads_started":1}
I20260927 13:56:20.147289 28307 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bbf161413f45bb8a7faf5432cc6622" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44509 } }
I20260927 13:56:20.147666 28307 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:20.147774 28307 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4bbf161413f45bb8a7faf5432cc6622, State: Initialized, Role: FOLLOWER
I20260927 13:56:20.148401 28307 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bbf161413f45bb8a7faf5432cc6622" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44509 } }
I20260927 13:56:20.148645 28307 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:56:20.148794 28307 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:56:20.148965 28307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:20.151067 28307 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bbf161413f45bb8a7faf5432cc6622" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44509 } }
I20260927 13:56:20.151613 28307 leader_election.cc:302] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4bbf161413f45bb8a7faf5432cc6622; no voters:
I20260927 13:56:20.152173 28307 leader_election.cc:288] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:56:20.152356 28312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:20.153056 28312 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [term 1 LEADER]: Becoming Leader. State: Replica: d4bbf161413f45bb8a7faf5432cc6622, State: Running, Role: LEADER
I20260927 13:56:20.153661 28312 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bbf161413f45bb8a7faf5432cc6622" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44509 } }
I20260927 13:56:20.154178 28307 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:20.156760 28314 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4bbf161413f45bb8a7faf5432cc6622. Latest consensus state: current_term: 1 leader_uuid: "d4bbf161413f45bb8a7faf5432cc6622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bbf161413f45bb8a7faf5432cc6622" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44509 } } }
I20260927 13:56:20.156718 28313 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4bbf161413f45bb8a7faf5432cc6622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bbf161413f45bb8a7faf5432cc6622" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44509 } } }
I20260927 13:56:20.157166 28313 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:20.158459 28314 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:20.159332 28319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:20.164702 28319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:20.174440 28319 catalog_manager.cc:1347] Generated new cluster ID: b882c6ccaf524b919db2482c9ae13a07
I20260927 13:56:20.174578 28319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:20.185524 28319 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:56:20.186806 28319 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:20.193498 28319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4bbf161413f45bb8a7faf5432cc6622: Generated new TSK 0
I20260927 13:56:20.194331 28319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:56:20.248986 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:44509
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:20.575163 28333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:20.575441 28333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:20.575624 28333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:20.582973 28333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:20.583123 28333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:20.583303 28333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:56:20.592851 28333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:44509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:20.594125 28333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:20.595458 28333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:20.608548 28339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:20.608596 28341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:20.608637 28338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:20.609616 28333 server_base.cc:1034] running on GCE node
I20260927 13:56:20.610265 28333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:20.611238 28333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:20.612473 28333 hybrid_clock.cc:648] HybridClock initialized: now 1790517380612415 us; error 42 us; skew 500 ppm
I20260927 13:56:20.612823 28333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:20 dist-test-slave-gkks krb5kdc[28227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517380, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:20.616001 28333 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:56:20.617401 28333 webserver.cc:460] Webserver started at http://127.22.26.193:36201/ using document root <none> and password file <none>
I20260927 13:56:20.618079 28333 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:20.618222 28333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:20.618522 28333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:20.621392 28333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "dff94bca46e84714bd31ff0790ba455f"
format_stamp: "Formatted at 2026-09-27 13:56:20 on dist-test-slave-gkks"
server_key: "b317a0c88dcad4e18569b2fac7bef38d"
server_key_iv: "f5fe4679760f561eceb8244c9db158c6"
server_key_version: "encryptionkey@0"
I20260927 13:56:20.622202 28333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "dff94bca46e84714bd31ff0790ba455f"
format_stamp: "Formatted at 2026-09-27 13:56:20 on dist-test-slave-gkks"
server_key: "b317a0c88dcad4e18569b2fac7bef38d"
server_key_iv: "f5fe4679760f561eceb8244c9db158c6"
server_key_version: "encryptionkey@0"
I20260927 13:56:20.627278 28333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 13:56:20.630558 28348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:20.632140 28333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:20.632319 28333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "dff94bca46e84714bd31ff0790ba455f"
format_stamp: "Formatted at 2026-09-27 13:56:20 on dist-test-slave-gkks"
server_key: "b317a0c88dcad4e18569b2fac7bef38d"
server_key_iv: "f5fe4679760f561eceb8244c9db158c6"
server_key_version: "encryptionkey@0"
I20260927 13:56:20.632535 28333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:20.668937 28333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:20.673429 28333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:20.673728 28333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:20.674695 28333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:20.676030 28333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:20.676139 28333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:20.676285 28333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:20.676355 28333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:20.709719 28333 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:38533
I20260927 13:56:20.709837 28460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:38533 every 8 connection(s)
I20260927 13:56:20.711037 28333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:56:20.720858 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 28333
I20260927 13:56:20.721113 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:56:20.721767 22635 external_mini_cluster.cc:1428] Setting key 993d8ae2a7e0fecbaf4398d0ed94d9a7
Sep 27 13:56:20 dist-test-slave-gkks krb5kdc[28227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517380, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:20.730789 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:20.713763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:47661 (local address 127.22.26.254:44509)
0927 13:56:20.714209 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:20.714214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:20.715217 (+ 1003us) server_negotiation.cc:407] Connection header received
0927 13:56:20.716979 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:20.716983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:20.717063 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:20.717181 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:20.718675 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:20.719518 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:20.721112 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:20.721547 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:20.721684 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:20.725247 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:20.725252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:20.725255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:20.725285 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:20.727855 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:20.728564 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:20.728571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:20.728575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:20.728696 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:20.729101 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:20.729107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:20.729111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:20.729373 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:20.729526 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:20.730228 (+ 702us) server_negotiation.cc:299] Negotiation successful
0927 13:56:20.730446 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":220,"threads_started":1}
I20260927 13:56:20.731838 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:20.714352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44509 (local address 127.22.26.193:47661)
0927 13:56:20.715043 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:20.715102 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:20.716705 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:20.717418 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:20.717426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:20.717946 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:20.718501 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:20.718518 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:20.719688 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:20.719698 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:20.720920 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:20.720933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:20.721794 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:20.721801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:20.722020 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:20.722701 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:20.722723 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:20.725025 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:20.727995 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:20.728003 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:20.728019 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:20.728447 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:20.728805 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:20.728810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:20.728814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:20.728990 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:20.729500 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:20.729508 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:20.729844 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 13:56:20.730103 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 13:56:20.730601 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":281,"threads_started":1}
I20260927 13:56:20.733366 28461 heartbeater.cc:347] Connected to a master server at 127.22.26.254:44509
I20260927 13:56:20.733656 28461 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:20.734346 28461 heartbeater.cc:511] Master 127.22.26.254:44509 requested a full tablet report, sending...
I20260927 13:56:20.736655 28272 ts_manager.cc:194] Registered new tserver with Master: dff94bca46e84714bd31ff0790ba455f (127.22.26.193:38533)
I20260927 13:56:20.738952 28272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:47661
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:56:20.770265 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:44509
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:21.091799 28467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:21.092082 28467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:21.092177 28467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:21.099377 28467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:21.099483 28467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:21.099596 28467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:56:21.109053 28467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:44509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:21.110283 28467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:21.111557 28467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:21.124221 28472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:21.124183 28473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:21.124882 28475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:21.125255 28467 server_base.cc:1034] running on GCE node
I20260927 13:56:21.125821 28467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:21.126807 28467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:21.127995 28467 hybrid_clock.cc:648] HybridClock initialized: now 1790517381127947 us; error 43 us; skew 500 ppm
I20260927 13:56:21.128327 28467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:21.131739 28467 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:56:21.133117 28467 webserver.cc:460] Webserver started at http://127.22.26.194:46397/ using document root <none> and password file <none>
I20260927 13:56:21.133788 28467 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:21.133936 28467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:21.134245 28467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:21.137142 28467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "a7ba21f0281047ebbce77505f2061c18"
format_stamp: "Formatted at 2026-09-27 13:56:21 on dist-test-slave-gkks"
server_key: "66f6895223daf3f7e6abd0ae194dcfb7"
server_key_iv: "087c3a4437b55a279f673de5521855c7"
server_key_version: "encryptionkey@0"
I20260927 13:56:21.137931 28467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "a7ba21f0281047ebbce77505f2061c18"
format_stamp: "Formatted at 2026-09-27 13:56:21 on dist-test-slave-gkks"
server_key: "66f6895223daf3f7e6abd0ae194dcfb7"
server_key_iv: "087c3a4437b55a279f673de5521855c7"
server_key_version: "encryptionkey@0"
I20260927 13:56:21.143119 28467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 13:56:21.146423 28482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:21.147771 28467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:21.147940 28467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "a7ba21f0281047ebbce77505f2061c18"
format_stamp: "Formatted at 2026-09-27 13:56:21 on dist-test-slave-gkks"
server_key: "66f6895223daf3f7e6abd0ae194dcfb7"
server_key_iv: "087c3a4437b55a279f673de5521855c7"
server_key_version: "encryptionkey@0"
I20260927 13:56:21.148144 28467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:21.171392 28467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:21.175690 28467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:21.175983 28467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:21.176970 28467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:21.178350 28467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:21.178452 28467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:21.178601 28467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:21.178671 28467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:21.213172 28467 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:38279
I20260927 13:56:21.213213 28594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:38279 every 8 connection(s)
I20260927 13:56:21.214498 28467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:56:21.219846 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 28467
I20260927 13:56:21.220038 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:56:21.220479 22635 external_mini_cluster.cc:1428] Setting key 4cdca37809f0d9ddcc81fa843367e59d
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:21.236235 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:21.217185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:40615 (local address 127.22.26.254:44509)
0927 13:56:21.217395 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:21.217402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:21.218697 (+ 1295us) server_negotiation.cc:407] Connection header received
0927 13:56:21.220717 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:21.220723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:21.220825 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:21.220940 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:21.222770 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.223535 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.225077 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.225502 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.225648 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:21.229227 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:21.229236 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:21.229245 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:21.229292 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:21.232912 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.233620 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.233628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.233635 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.233793 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.234193 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.234201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.234208 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.234601 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:21.234759 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:21.235475 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 13:56:21.235744 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:56:21.236994 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:21.217771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44509 (local address 127.22.26.194:40615)
0927 13:56:21.218519 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:21.218563 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:21.220404 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:21.221186 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:21.221194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:21.221929 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:21.222588 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:21.222611 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.223699 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.223704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:21.224928 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:21.224940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.225723 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.225730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:21.225971 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:21.226700 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:21.226723 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:21.228982 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:21.233045 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:21.233052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:21.233065 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:21.233493 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:21.233904 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:21.233907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:21.233909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:21.234084 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:21.234720 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:21.234726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:21.235035 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 13:56:21.235344 (+ 309us) client_negotiation.cc:240] Negotiation successful
0927 13:56:21.235737 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":310,"threads_started":1}
I20260927 13:56:21.238497 28595 heartbeater.cc:347] Connected to a master server at 127.22.26.254:44509
I20260927 13:56:21.238823 28595 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:21.239609 28595 heartbeater.cc:511] Master 127.22.26.254:44509 requested a full tablet report, sending...
I20260927 13:56:21.241050 28272 ts_manager.cc:194] Registered new tserver with Master: a7ba21f0281047ebbce77505f2061c18 (127.22.26.194:38279)
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
I20260927 13:56:21.241935 28272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:40615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:56:21.271138 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:44509
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:21.612087 28600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:21.612358 28600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:21.612452 28600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:21.619632 28600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:21.619735 28600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:21.619844 28600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:56:21.629140 28600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:44509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:21.630326 28600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:21.631575 28600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:21.644495 28608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:21.644546 28605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:21.644582 28606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:21.645610 28600 server_base.cc:1034] running on GCE node
I20260927 13:56:21.646302 28600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:21.647295 28600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:21.648546 28600 hybrid_clock.cc:648] HybridClock initialized: now 1790517381648474 us; error 62 us; skew 500 ppm
I20260927 13:56:21.648873 28600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:21.652117 28600 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:56:21.653563 28600 webserver.cc:460] Webserver started at http://127.22.26.195:35991/ using document root <none> and password file <none>
I20260927 13:56:21.654277 28600 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:21.654418 28600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:21.654726 28600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:21.657653 28600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "c334939cf77e4ec38fa165e2f6f10da4"
format_stamp: "Formatted at 2026-09-27 13:56:21 on dist-test-slave-gkks"
server_key: "5a1e1e898895580c30c2981bacc3412b"
server_key_iv: "4a6fea102f3a009a39ad9c5faa6d3524"
server_key_version: "encryptionkey@0"
I20260927 13:56:21.658447 28600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "c334939cf77e4ec38fa165e2f6f10da4"
format_stamp: "Formatted at 2026-09-27 13:56:21 on dist-test-slave-gkks"
server_key: "5a1e1e898895580c30c2981bacc3412b"
server_key_iv: "4a6fea102f3a009a39ad9c5faa6d3524"
server_key_version: "encryptionkey@0"
I20260927 13:56:21.663807 28600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 13:56:21.667555 28615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:21.668816 28600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 13:56:21.668978 28600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "c334939cf77e4ec38fa165e2f6f10da4"
format_stamp: "Formatted at 2026-09-27 13:56:21 on dist-test-slave-gkks"
server_key: "5a1e1e898895580c30c2981bacc3412b"
server_key_iv: "4a6fea102f3a009a39ad9c5faa6d3524"
server_key_version: "encryptionkey@0"
I20260927 13:56:21.669176 28600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:21.685452 28600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:21.689292 28600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:21.689565 28600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:21.690495 28600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:21.691826 28600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:21.691895 28600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:21.692061 28600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:21.692109 28600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:21.725221 28600 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:36079
I20260927 13:56:21.725270 28727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:36079 every 8 connection(s)
I20260927 13:56:21.726445 28600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:56:21.730302 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 28600
I20260927 13:56:21.730508 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:56:21.731061 22635 external_mini_cluster.cc:1428] Setting key 703434a3a2bf72261ae8b23186e96b01
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:21.741851 28461 heartbeater.cc:503] Master 127.22.26.254:44509 was elected leader, sending a full tablet report...
I20260927 13:56:21.747071 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:21.728937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:51601 (local address 127.22.26.254:44509)
0927 13:56:21.729117 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:21.729122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:21.730419 (+ 1297us) server_negotiation.cc:407] Connection header received
0927 13:56:21.732640 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:21.732644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:21.732738 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:21.732817 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:21.734663 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.735504 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.737099 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.737547 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.737640 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:21.741009 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:21.741014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:21.741017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:21.741047 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:21.744035 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.744680 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.744685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.744688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.744839 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.745274 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.745280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.745284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.745665 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:21.745845 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:21.746510 (+ 665us) server_negotiation.cc:299] Negotiation successful
0927 13:56:21.746721 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:56:21.748080 28730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:21.729409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44509 (local address 127.22.26.195:51601)
0927 13:56:21.730213 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:21.730272 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:21.732371 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:21.733059 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:21.733067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:21.733807 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:21.734512 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:21.734531 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.735638 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.735642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:21.736979 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:21.736990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.737770 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.737780 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:21.737996 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:21.738679 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:21.738701 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:21.740826 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:21.744154 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:21.744160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:21.744171 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:21.744562 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:21.744958 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:21.744964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:21.744968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:21.745147 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:21.745815 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:21.745823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:21.746146 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 13:56:21.746421 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 13:56:21.746912 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":296,"threads_started":1}
I20260927 13:56:21.749392 28728 heartbeater.cc:347] Connected to a master server at 127.22.26.254:44509
I20260927 13:56:21.749650 28728 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:21.750293 28728 heartbeater.cc:511] Master 127.22.26.254:44509 requested a full tablet report, sending...
I20260927 13:56:21.751838 28272 ts_manager.cc:194] Registered new tserver with Master: c334939cf77e4ec38fa165e2f6f10da4 (127.22.26.195:36079)
I20260927 13:56:21.752619 28272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:51601
I20260927 13:56:21.758531 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/test-admin.keytab.
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:21.785638 22635 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 13:56:21.791328 28735 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:21.788483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.22.26.193:38533)
0927 13:56:21.789174 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:21.789181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:21.789208 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:56:21.789272 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:21.789278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:21.789512 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:21.789713 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:21.790878 (+ 1165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40260: BlockingRecv error: recv got EOF from 127.0.0.1:40260 (error 108)
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":366,"threads_started":1}
W20260927 13:56:21.791391 22635 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/krb5cc' of type FILE
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:21.793273 22635 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:21.794832 22635 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:56:21.810655 28735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:21.796595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.22.26.193:38533)
0927 13:56:21.796887 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:21.796893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:21.797064 (+ 171us) server_negotiation.cc:407] Connection header received
0927 13:56:21.797224 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:21.797228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:21.797289 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:21.797373 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:21.798374 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.799495 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.800741 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.801292 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.801389 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:21.805075 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:21.805093 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:21.805115 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:21.805168 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:21.808158 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.808812 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.808819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.808821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.808941 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.809250 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.809253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.809255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.809589 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:21.809819 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:21.810104 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 13:56:21.810369 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260927 13:56:21.816530 28735 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:21.814734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.22.26.193:38533)
0927 13:56:21.815020 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:21.815026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:21.815347 (+ 321us) server_negotiation.cc:407] Connection header received
0927 13:56:21.815471 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:21.815474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:21.815529 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:21.815606 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:21.816320 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40278: BlockingRecv error: recv got EOF from 127.0.0.1:40278 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 13:56:21.816696 22635 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/krb5cc' of type FILE
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:21.818344 22635 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:21.819628 22635 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:56:21.832886 28735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:21.820947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.22.26.193:38533)
0927 13:56:21.821132 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:21.821138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:21.821259 (+ 121us) server_negotiation.cc:407] Connection header received
0927 13:56:21.821370 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:21.821373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:21.821423 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:21.821499 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:21.822473 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.823209 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.824445 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.824853 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.824941 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:21.828281 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:21.828286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:21.828289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:21.828344 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:21.830740 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.831357 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.831361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.831364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.831478 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.831805 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.831807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.831809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.832034 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:21.832221 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:21.832434 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 13:56:21.832600 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 13:56:21.837173 28735 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:21.835407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.22.26.193:38533)
0927 13:56:21.835580 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:21.835584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:21.835924 (+ 340us) server_negotiation.cc:407] Connection header received
0927 13:56:21.836014 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:21.836017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:21.836080 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:21.836173 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:21.836876 (+ 703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40300: BlockingRecv error: recv got EOF from 127.0.0.1:40300 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 13:56:21.837285 22635 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790517330258165-22635-0/krb5kdc/krb5cc' of type FILE
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:21.838783 22635 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:21.840193 22635 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:56:21.855340 28735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:21.841486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.22.26.193:38533)
0927 13:56:21.841772 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:21.841780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:21.842092 (+ 312us) server_negotiation.cc:407] Connection header received
0927 13:56:21.842327 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:21.842331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:21.842389 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:21.842499 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:21.843629 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.844412 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.845786 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.846267 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.846414 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:21.850121 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:21.850126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:21.850129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:21.850162 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:21.852955 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.853531 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.853535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.853537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.853682 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.854049 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.854052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.854054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.854301 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:21.854516 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:21.854805 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 13:56:21.854985 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260927 13:56:21.856854 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 28333
I20260927 13:56:21.868430 28735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:21.857553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40312 (local address 127.22.26.193:38533)
0927 13:56:21.857705 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:21.857708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:21.858073 (+ 365us) server_negotiation.cc:407] Connection header received
0927 13:56:21.858202 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:21.858205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:21.858259 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:21.858334 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:21.859252 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.859998 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.861162 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.861563 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.861655 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:21.863650 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:21.863655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:21.863658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:21.863686 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:21.866119 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.866705 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.866709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.866711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.866844 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.867244 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.867248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.867250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.867480 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:21.867669 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:21.867981 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 13:56:21.868153 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:56:21.869068 28456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:21.954591 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 28333
I20260927 13:56:21.970723 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 28467
Sep 27 13:56:21 dist-test-slave-gkks krb5kdc[28227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:56:21.986581 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:21.971704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.22.26.194:38279)
0927 13:56:21.972227 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:21.972233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:21.972259 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:56:21.972318 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:21.972324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:21.972533 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:21.972700 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:21.973760 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.974755 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.976187 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:21.976765 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:21.976903 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:21.980910 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:21.980920 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:21.980935 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:21.980975 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:21.983766 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.984489 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.984496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.984498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.984623 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:21.985005 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:21.985034 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:21.985037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:21.985375 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:21.985687 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:21.986013 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 13:56:21.986272 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":270,"threads_started":1}
I20260927 13:56:21.988093 28590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:22.070288 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 28467
I20260927 13:56:22.086297 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 28600
Sep 27 13:56:22 dist-test-slave-gkks krb5kdc[28227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:56:22.102434 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:22.087418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.22.26.195:36079)
0927 13:56:22.088080 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:22.088087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:22.088116 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:56:22.088180 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:22.088187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:22.088427 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:22.088563 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:22.089552 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:22.090567 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:22.092128 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:22.092685 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:22.092787 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:22.096839 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:22.096851 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:22.096864 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:22.096899 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:22.099640 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:22.100351 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:22.100358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:22.100360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:22.100488 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:22.100889 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:22.100894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:22.100896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:22.101225 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:22.101574 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:22.101867 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 13:56:22.102121 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":354,"threads_started":1}
I20260927 13:56:22.103770 28723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:22.186139 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 28600
I20260927 13:56:22.201850 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 28240
I20260927 13:56:22.203198 28302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:22.306159 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 28240
2026-09-27T13:56:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3897 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:56:22 dist-test-slave-gkks krb5kdc[28755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:56:22 dist-test-slave-gkks krb5kdc[28755](info): set up 2 sockets
Sep 27 13:56:22 dist-test-slave-gkks krb5kdc[28755](info): commencing operation
krb5kdc: starting...
W20260927 13:56:23.396392 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.008s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:56:23 dist-test-slave-gkks krb5kdc[28755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:56:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:56:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.254.keytab.
Entry for principal oryx/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.254.keytab.
I20260927 13:56:23.594424 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39559
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.254.keytab
--principal=oryx/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:39559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:23.911614 28768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:23.911964 28768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:23.912130 28768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:23.919399 28768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:23.919540 28768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:23.919647 28768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:23.919749 28768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:23.919824 28768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:56:23.930068 28768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:39559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.254.keytab
--principal=oryx/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:23.931711 28768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:23.933033 28768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:23.944805 28773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:23.944766 28776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:23.944794 28774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:23.945755 28768 server_base.cc:1034] running on GCE node
I20260927 13:56:23.946476 28768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:23.947975 28768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:23.949224 28768 hybrid_clock.cc:648] HybridClock initialized: now 1790517383949179 us; error 53 us; skew 500 ppm
I20260927 13:56:23.949587 28768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:23 dist-test-slave-gkks krb5kdc[28755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517383, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:23.959558 28768 init.cc:377] Logged in from keytab as oryx/127.22.26.254@KRBTEST.COM (short username oryx)
I20260927 13:56:23.961150 28768 webserver.cc:460] Webserver started at http://127.22.26.254:40829/ using document root <none> and password file <none>
I20260927 13:56:23.961869 28768 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:23.962023 28768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:23.962340 28768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:23.965406 28768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "ae445534f86344d9bf1ef1b1e0328b91"
format_stamp: "Formatted at 2026-09-27 13:56:23 on dist-test-slave-gkks"
server_key: "d0f89d4d0e7479196873d847c1e5e1d5"
server_key_iv: "9d6631aeaba499b715f4e7bc6d1045ee"
server_key_version: "encryptionkey@0"
I20260927 13:56:23.966172 28768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "ae445534f86344d9bf1ef1b1e0328b91"
format_stamp: "Formatted at 2026-09-27 13:56:23 on dist-test-slave-gkks"
server_key: "d0f89d4d0e7479196873d847c1e5e1d5"
server_key_iv: "9d6631aeaba499b715f4e7bc6d1045ee"
server_key_version: "encryptionkey@0"
I20260927 13:56:23.971480 28768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 13:56:23.974804 28783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:23.976300 28768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:23.976478 28768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "ae445534f86344d9bf1ef1b1e0328b91"
format_stamp: "Formatted at 2026-09-27 13:56:23 on dist-test-slave-gkks"
server_key: "d0f89d4d0e7479196873d847c1e5e1d5"
server_key_iv: "9d6631aeaba499b715f4e7bc6d1045ee"
server_key_version: "encryptionkey@0"
I20260927 13:56:23.976675 28768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:23.999475 28768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:24.003978 28768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:24.004289 28768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:24.022713 28768 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:39559
I20260927 13:56:24.022748 28834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:39559 every 8 connection(s)
I20260927 13:56:24.024103 28768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:56:24.027820 28835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:24.032248 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 28768
I20260927 13:56:24.032512 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:56:24.033324 22635 external_mini_cluster.cc:1428] Setting key fad2b767245e53334259f26debcfcbff
I20260927 13:56:24.039598 28835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91: Bootstrap starting.
I20260927 13:56:24.043180 28835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:24.044322 28835 log.cc:824] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:24.047829 28835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91: No bootstrap required, opened a new log
Sep 27 13:56:24 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
I20260927 13:56:24.058633 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:24.035450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51064 (local address 127.22.26.254:39559)
0927 13:56:24.036268 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:24.036279 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:24.036321 (+ 42us) server_negotiation.cc:407] Connection header received
0927 13:56:24.038045 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:24.038066 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:24.038439 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:24.038753 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:24.040588 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:24.041930 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:24.043791 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:24.044545 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:24.044686 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:24.049557 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:24.049572 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:24.049590 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:24.049643 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:24.053874 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:24.054594 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:24.054603 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:24.054613 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:24.054824 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:24.055304 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:24.055310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:24.055313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:24.056028 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:24.056364 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:24.056783 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 13:56:24.057281 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":398,"threads_started":1}
I20260927 13:56:24.058663 28835 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } }
I20260927 13:56:24.059121 28835 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:24.059238 28835 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae445534f86344d9bf1ef1b1e0328b91, State: Initialized, Role: FOLLOWER
I20260927 13:56:24.059831 28835 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } }
I20260927 13:56:24.060089 28835 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:56:24.060238 28835 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:56:24.060441 28835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:24.062688 28835 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } }
I20260927 13:56:24.063246 28835 leader_election.cc:302] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae445534f86344d9bf1ef1b1e0328b91; no voters:
I20260927 13:56:24.063849 28835 leader_election.cc:288] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:56:24.064023 28840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:24.065230 28840 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [term 1 LEADER]: Becoming Leader. State: Replica: ae445534f86344d9bf1ef1b1e0328b91, State: Running, Role: LEADER
I20260927 13:56:24.065714 28840 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } }
I20260927 13:56:24.066025 28835 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:24.068401 28842 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae445534f86344d9bf1ef1b1e0328b91. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } }
I20260927 13:56:24.068835 28842 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:24.069437 28841 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } }
I20260927 13:56:24.070271 28841 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:24.073288 28849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:24.078167 28849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:24.086934 28849 catalog_manager.cc:1347] Generated new cluster ID: 592198b8ff06425389edd11acf820348
I20260927 13:56:24.087055 28849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:24.101938 28849 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:56:24.103255 28849 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:24.110466 28849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91: Generated new TSK 0
I20260927 13:56:24.111436 28849 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab.
Entry for principal oryx/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab.
I20260927 13:56:24.178520 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39559
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab
--principal=oryx/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:24.513734 28861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:24.513994 28861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:24.514092 28861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:24.521132 28861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:24.521281 28861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:24.521420 28861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:56:24.530658 28861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab
--principal=oryx/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:24.531908 28861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:24.533154 28861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:24.546461 28866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:24.546331 28869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:24.546402 28867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:24.547482 28861 server_base.cc:1034] running on GCE node
I20260927 13:56:24.548096 28861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:24.548952 28861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:24.550184 28861 hybrid_clock.cc:648] HybridClock initialized: now 1790517384550146 us; error 52 us; skew 500 ppm
I20260927 13:56:24.550500 28861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:24 dist-test-slave-gkks krb5kdc[28755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517384, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:24.553649 28861 init.cc:377] Logged in from keytab as oryx/127.22.26.193@KRBTEST.COM (short username oryx)
I20260927 13:56:24.555016 28861 webserver.cc:460] Webserver started at http://127.22.26.193:46135/ using document root <none> and password file <none>
I20260927 13:56:24.555704 28861 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:24.555805 28861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:24.556058 28861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:24.558982 28861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "88dc53435f9d4db09dc2b185d4505baf"
format_stamp: "Formatted at 2026-09-27 13:56:24 on dist-test-slave-gkks"
server_key: "d5f6ed738b8dd1a12eaed04f33015db2"
server_key_iv: "00b69140478730c41792e492f8bf13b0"
server_key_version: "encryptionkey@0"
I20260927 13:56:24.559818 28861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "88dc53435f9d4db09dc2b185d4505baf"
format_stamp: "Formatted at 2026-09-27 13:56:24 on dist-test-slave-gkks"
server_key: "d5f6ed738b8dd1a12eaed04f33015db2"
server_key_iv: "00b69140478730c41792e492f8bf13b0"
server_key_version: "encryptionkey@0"
I20260927 13:56:24.564911 28861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:56:24.568445 28876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:24.569868 28861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:56:24.570032 28861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "88dc53435f9d4db09dc2b185d4505baf"
format_stamp: "Formatted at 2026-09-27 13:56:24 on dist-test-slave-gkks"
server_key: "d5f6ed738b8dd1a12eaed04f33015db2"
server_key_iv: "00b69140478730c41792e492f8bf13b0"
server_key_version: "encryptionkey@0"
I20260927 13:56:24.570205 28861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:24.597038 28861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:24.601295 28861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:24.601585 28861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:24.602563 28861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:24.603956 28861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:24.604054 28861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:24.604199 28861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:24.604269 28861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:24.636345 28861 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:43167
I20260927 13:56:24.636431 28988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:43167 every 8 connection(s)
I20260927 13:56:24.637609 28861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:56:24.643414 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 28861
I20260927 13:56:24.643658 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:56:24.644484 22635 external_mini_cluster.cc:1428] Setting key ffdcc759a1a7fb8b0484fa65192b7798
Sep 27 13:56:24 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517384, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
I20260927 13:56:24.657752 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:24.640141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:60347 (local address 127.22.26.254:39559)
0927 13:56:24.640606 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:24.640611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:24.641369 (+ 758us) server_negotiation.cc:407] Connection header received
0927 13:56:24.643207 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:24.643212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:24.643288 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:24.643382 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:24.645418 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:24.646432 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:24.648033 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:24.648451 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:24.648594 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:24.652211 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:24.652216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:24.652219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:24.652250 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:24.654699 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:24.655507 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:24.655511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:24.655514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:24.655628 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:24.656059 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:24.656064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:24.656067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:24.656330 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:24.656486 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:24.657178 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 13:56:24.657400 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":256,"threads_started":1}
I20260927 13:56:24.658670 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:24.640544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.193:60347)
0927 13:56:24.641209 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:24.641252 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:24.642858 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:24.643688 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:24.643702 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:24.644452 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:24.645236 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:24.645257 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:24.646575 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:24.646587 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:24.647882 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:24.647894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:24.648690 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:24.648698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:24.648923 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:24.649640 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:24.649661 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:24.651997 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:24.654858 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:24.654866 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:24.654882 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:24.655373 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:24.655739 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:24.655746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:24.655751 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:24.655931 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:24.656474 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:24.656482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:24.656788 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 13:56:24.657042 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 13:56:24.657548 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":259,"threads_started":1}
I20260927 13:56:24.660252 28989 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39559
I20260927 13:56:24.660583 28989 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:24.661510 28989 heartbeater.cc:511] Master 127.22.26.254:39559 requested a full tablet report, sending...
I20260927 13:56:24.664011 28800 ts_manager.cc:194] Registered new tserver with Master: 88dc53435f9d4db09dc2b185d4505baf (127.22.26.193:43167)
WARNING: no policy specified for oryx/127.22.26.194@KRBTEST.COM; defaulting to no policy
I20260927 13:56:24.665789 28800 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:60347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab.
Entry for principal oryx/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab.
I20260927 13:56:24.690400 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39559
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab
--principal=oryx/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:25.005638 28995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:25.005905 28995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:25.006060 28995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:25.013248 28995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:25.013360 28995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:25.013573 28995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:56:25.023209 28995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab
--principal=oryx/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:25.024459 28995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:25.025758 28995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:25.039067 29003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:25.039155 29001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:25.039177 29000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:25.039919 28995 server_base.cc:1034] running on GCE node
I20260927 13:56:25.040494 28995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:25.041405 28995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:25.042637 28995 hybrid_clock.cc:648] HybridClock initialized: now 1790517385042589 us; error 61 us; skew 500 ppm
I20260927 13:56:25.042970 28995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:25 dist-test-slave-gkks krb5kdc[28755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517385, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:25.046070 28995 init.cc:377] Logged in from keytab as oryx/127.22.26.194@KRBTEST.COM (short username oryx)
I20260927 13:56:25.047431 28995 webserver.cc:460] Webserver started at http://127.22.26.194:41073/ using document root <none> and password file <none>
I20260927 13:56:25.048018 28995 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:25.048121 28995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:25.048364 28995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:25.050993 28995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "26964d7eab3b4d0f8940d3e911c911ac"
format_stamp: "Formatted at 2026-09-27 13:56:25 on dist-test-slave-gkks"
server_key: "0e019099679f978af666c9757b2f1be3"
server_key_iv: "800b4cd07ecfb9810a1b5caf936a66fa"
server_key_version: "encryptionkey@0"
I20260927 13:56:25.051761 28995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "26964d7eab3b4d0f8940d3e911c911ac"
format_stamp: "Formatted at 2026-09-27 13:56:25 on dist-test-slave-gkks"
server_key: "0e019099679f978af666c9757b2f1be3"
server_key_iv: "800b4cd07ecfb9810a1b5caf936a66fa"
server_key_version: "encryptionkey@0"
I20260927 13:56:25.056703 28995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:56:25.060218 29010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:25.061468 28995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:56:25.061596 28995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "26964d7eab3b4d0f8940d3e911c911ac"
format_stamp: "Formatted at 2026-09-27 13:56:25 on dist-test-slave-gkks"
server_key: "0e019099679f978af666c9757b2f1be3"
server_key_iv: "800b4cd07ecfb9810a1b5caf936a66fa"
server_key_version: "encryptionkey@0"
I20260927 13:56:25.061756 28995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:25.079576 28995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:25.083616 28995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:25.083894 28995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:25.084743 28995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:25.085988 28995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:25.086068 28995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:25.086241 28995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:25.086289 28995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:25.116819 28995 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:44945
I20260927 13:56:25.116860 29122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:44945 every 8 connection(s)
I20260927 13:56:25.118055 28995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:56:25.118511 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 28995
I20260927 13:56:25.118741 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:56:25.119359 22635 external_mini_cluster.cc:1428] Setting key 242bbab34db5bda0dc4ce35f510531c9
Sep 27 13:56:25 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517385, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.22.26.195@KRBTEST.COM; defaulting to no policy
I20260927 13:56:25.139387 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:25.120714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:54949 (local address 127.22.26.254:39559)
0927 13:56:25.120890 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:25.120896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:25.122043 (+ 1147us) server_negotiation.cc:407] Connection header received
0927 13:56:25.123838 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:25.123844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:25.123935 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:25.124042 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:25.125473 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:25.126184 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:25.128440 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:25.128880 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:25.128985 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:25.133083 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:25.133089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:25.133093 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:25.133129 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:25.136159 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:25.136917 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:25.136924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:25.136936 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:25.137069 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:25.137535 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:25.137554 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:25.137556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:25.137834 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:25.138005 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:25.138723 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 13:56:25.138961 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:56:25.140391 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:25.121197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.194:54949)
0927 13:56:25.121895 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:25.121937 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:25.123579 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:25.124276 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:25.124284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:25.124775 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:25.125314 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:25.125331 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:25.126343 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:25.126347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:25.128277 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:25.128294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:25.129102 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:25.129111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:25.129443 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:25.130141 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:25.130179 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:25.132837 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:25.136311 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:25.136320 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:25.136336 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:25.136778 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:25.137191 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:25.137198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:25.137202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:25.137401 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:25.137974 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:25.137984 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:25.138338 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 13:56:25.138621 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 13:56:25.139185 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":287,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.26.195@KRBTEST.COM" created.
I20260927 13:56:25.142098 29123 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39559
I20260927 13:56:25.142421 29123 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:25.143218 29123 heartbeater.cc:511] Master 127.22.26.254:39559 requested a full tablet report, sending...
I20260927 13:56:25.144685 28800 ts_manager.cc:194] Registered new tserver with Master: 26964d7eab3b4d0f8940d3e911c911ac (127.22.26.194:44945)
I20260927 13:56:25.145485 28800 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:54949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab.
Entry for principal oryx/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab.
I20260927 13:56:25.166260 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39559
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab
--principal=oryx/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:25.483204 29128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:25.483454 29128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:25.483592 29128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:25.490612 29128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:25.490707 29128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:25.490863 29128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:56:25.500118 29128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab
--principal=oryx/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:25.501312 29128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:25.502591 29128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:25.514834 29133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:25.515055 29134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:25.516202 29136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:25.517086 29128 server_base.cc:1034] running on GCE node
I20260927 13:56:25.517642 29128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:25.518503 29128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:25.519721 29128 hybrid_clock.cc:648] HybridClock initialized: now 1790517385519692 us; error 41 us; skew 500 ppm
I20260927 13:56:25.520017 29128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:25 dist-test-slave-gkks krb5kdc[28755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517385, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:25.523068 29128 init.cc:377] Logged in from keytab as oryx/127.22.26.195@KRBTEST.COM (short username oryx)
I20260927 13:56:25.524607 29128 webserver.cc:460] Webserver started at http://127.22.26.195:38207/ using document root <none> and password file <none>
I20260927 13:56:25.525287 29128 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:25.525426 29128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:25.525712 29128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:25.528626 29128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5"
format_stamp: "Formatted at 2026-09-27 13:56:25 on dist-test-slave-gkks"
server_key: "c1e4c2c57de6b4cc524bfdefc9fab690"
server_key_iv: "f1adfd27f5952ee71dc2c29697f3b91b"
server_key_version: "encryptionkey@0"
I20260927 13:56:25.529398 29128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5"
format_stamp: "Formatted at 2026-09-27 13:56:25 on dist-test-slave-gkks"
server_key: "c1e4c2c57de6b4cc524bfdefc9fab690"
server_key_iv: "f1adfd27f5952ee71dc2c29697f3b91b"
server_key_version: "encryptionkey@0"
I20260927 13:56:25.534448 29128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 13:56:25.537787 29143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:25.539052 29128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:56:25.539264 29128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5"
format_stamp: "Formatted at 2026-09-27 13:56:25 on dist-test-slave-gkks"
server_key: "c1e4c2c57de6b4cc524bfdefc9fab690"
server_key_iv: "f1adfd27f5952ee71dc2c29697f3b91b"
server_key_version: "encryptionkey@0"
I20260927 13:56:25.539469 29128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:25.576469 29128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:25.580363 29128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:25.580647 29128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:25.581607 29128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:25.582927 29128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:25.583027 29128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:25.583200 29128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:25.583274 29128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:25.616875 29128 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:46789
I20260927 13:56:25.616909 29255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:46789 every 8 connection(s)
I20260927 13:56:25.618216 29128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:56:25.625208 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 29128
I20260927 13:56:25.625413 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:56:25.625962 22635 external_mini_cluster.cc:1428] Setting key ebcee8ef57cc9ee67861d7c5e3d09cba
Sep 27 13:56:25 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517385, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
I20260927 13:56:25.638972 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:25.620963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:50501 (local address 127.22.26.254:39559)
0927 13:56:25.621137 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:25.621141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:25.622185 (+ 1044us) server_negotiation.cc:407] Connection header received
0927 13:56:25.624026 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:25.624031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:25.624145 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:25.624268 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:25.626185 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:25.627304 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:25.629301 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:25.629740 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:25.629849 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:25.633082 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:25.633086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:25.633090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:25.633120 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:25.635849 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:25.636679 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:25.636683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:25.636685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:25.636806 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:25.637301 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:25.637305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:25.637307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:25.637608 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:25.637752 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:25.638529 (+ 777us) server_negotiation.cc:299] Negotiation successful
0927 13:56:25.638714 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:56:25.640247 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:25.621341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.195:50501)
0927 13:56:25.622022 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:25.622064 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:25.623766 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:25.624488 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:25.624496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:25.625147 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:25.625940 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:25.625966 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:25.627472 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:25.627477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:25.629145 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:25.629159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:25.629925 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:25.629929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:25.630110 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:25.630763 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:25.630788 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:25.632888 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:25.635996 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:25.636004 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:25.636018 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:25.636554 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:25.636917 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:25.636922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:25.636928 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:25.637160 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:25.637758 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:25.637765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:25.638145 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 13:56:25.638437 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 13:56:25.638940 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":279,"threads_started":1}
I20260927 13:56:25.641934 29256 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39559
I20260927 13:56:25.642204 29256 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:25.642915 29256 heartbeater.cc:511] Master 127.22.26.254:39559 requested a full tablet report, sending...
I20260927 13:56:25.644462 28799 ts_manager.cc:194] Registered new tserver with Master: 7b1eb4bf19e9413a8f3d63ff6b7583c5 (127.22.26.195:46789)
I20260927 13:56:25.645370 28799 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:50501
I20260927 13:56:25.654026 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:56:25.668650 28989 heartbeater.cc:503] Master 127.22.26.254:39559 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.22.26.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.26.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.26.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.253.keytab.
Entry for principal oryx/127.22.26.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.253.keytab.
I20260927 13:56:25.702575 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.253:45111
--webserver_interface=127.22.26.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.253.keytab
--principal=oryx/127.22.26.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:39559,127.22.26.253:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:26.023137 29261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:26.023422 29261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:26.023622 29261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:26.030759 29261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:26.030855 29261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:26.030964 29261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:26.031068 29261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:26.031170 29261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:56:26.040796 29261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:39559,127.22.26.253:45111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.253.keytab
--principal=oryx/127.22.26.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.253:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.26.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
W20260927 13:56:26.041167 29261 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.22.26.254:39559,127.22.26.253:45111'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 13:56:26.042209 29261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:26.043673 29261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:26.055164 29269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:26.055146 29266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:26.055282 29267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:26.060549 29261 server_base.cc:1034] running on GCE node
I20260927 13:56:26.061353 29261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:26.062654 29261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:26.063900 29261 hybrid_clock.cc:648] HybridClock initialized: now 1790517386063839 us; error 58 us; skew 500 ppm
I20260927 13:56:26.064224 29261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:26 dist-test-slave-gkks krb5kdc[28755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517386, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:26.067549 29261 init.cc:377] Logged in from keytab as oryx/127.22.26.253@KRBTEST.COM (short username oryx)
I20260927 13:56:26.068908 29261 webserver.cc:460] Webserver started at http://127.22.26.253:46649/ using document root <none> and password file <none>
I20260927 13:56:26.069736 29261 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:26.069844 29261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:26.070159 29261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:26.073026 29261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/data/instance:
uuid: "d9f5a277ee88440eb96a393fe2489a5b"
format_stamp: "Formatted at 2026-09-27 13:56:26 on dist-test-slave-gkks"
server_key: "fedb42c0a1be1f85f33f6c8b95fcd94c"
server_key_iv: "6eacda647cc45b2148206236f17f78bd"
server_key_version: "encryptionkey@0"
I20260927 13:56:26.073863 29261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/wal/instance:
uuid: "d9f5a277ee88440eb96a393fe2489a5b"
format_stamp: "Formatted at 2026-09-27 13:56:26 on dist-test-slave-gkks"
server_key: "fedb42c0a1be1f85f33f6c8b95fcd94c"
server_key_iv: "6eacda647cc45b2148206236f17f78bd"
server_key_version: "encryptionkey@0"
I20260927 13:56:26.078822 29261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 13:56:26.082204 29276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:26.083570 29261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:26.083751 29261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/wal
uuid: "d9f5a277ee88440eb96a393fe2489a5b"
format_stamp: "Formatted at 2026-09-27 13:56:26 on dist-test-slave-gkks"
server_key: "fedb42c0a1be1f85f33f6c8b95fcd94c"
server_key_iv: "6eacda647cc45b2148206236f17f78bd"
server_key_version: "encryptionkey@0"
I20260927 13:56:26.083947 29261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:26.115967 29261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:26.120466 29261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:26.120728 29261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:26.138652 29261 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.253:45111
I20260927 13:56:26.138674 29327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.253:45111 every 8 connection(s)
I20260927 13:56:26.140125 29261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/data/info.pb
I20260927 13:56:26.141927 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 29261
I20260927 13:56:26.144270 29328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:26.148403 29123 heartbeater.cc:503] Master 127.22.26.254:39559 was elected leader, sending a full tablet report...
I20260927 13:56:26.153358 29328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:56:26 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.26.253@KRBTEST.COM
Sep 27 13:56:26 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517386, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.253@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
I20260927 13:56:26.168433 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.144496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.22.26.253:45111)
0927 13:56:26.145275 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.145286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.145326 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:56:26.146870 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.146889 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.147360 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.147681 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.149500 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.151053 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.152838 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.153468 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.153600 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.158844 (+ 5244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.158851 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.158859 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.158900 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.163605 (+ 4705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.164382 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.164388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.164393 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.164572 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.165031 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.165039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.165044 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.165742 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.165988 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.166577 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.166986 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":364,"threads_started":1}
I20260927 13:56:26.172896 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.156234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51080)
0927 13:56:26.156957 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.157006 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.157161 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.157950 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.157959 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.158826 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:26.159165 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.159175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.160667 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.160675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.161687 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.161698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.162707 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.162712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.162830 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.163476 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:26.163494 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:26.166276 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:26.169784 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.169794 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.169800 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.170345 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.170842 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.170850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.170854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.171055 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.171891 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:26.171898 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:26.172059 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.172232 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.172524 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":290,"threads_started":1}
I20260927 13:56:26.174006 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.156325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.22.26.254:39559)
0927 13:56:26.157606 (+ 1281us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.157614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.157634 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:56:26.157695 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.157700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.157776 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.157858 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.159320 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.160492 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.161848 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.162518 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.162652 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.166478 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.166484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.166489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.166528 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.169624 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.170487 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.170492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.170496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.170642 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.171245 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.171253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.171260 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.171654 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.171819 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.173191 (+ 1372us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.173523 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1165,"thread_start_us":1067,"threads_started":1}
I20260927 13:56:26.179943 29328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:56:26 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517386, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.253@KRBTEST.COM for oryx/127.22.26.253@KRBTEST.COM
I20260927 13:56:26.192154 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.180652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:45111 (local address 127.0.0.1:56822)
0927 13:56:26.180875 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.180894 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.180990 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.181259 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.181263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.181601 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:26.181719 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.181727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.182667 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.182672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.183588 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.183599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.184275 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.184280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.184470 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.184951 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:26.184967 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:26.186938 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:26.190106 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.190110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.190113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.190483 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.190819 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.190822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.190824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.190943 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.191453 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:26.191456 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:26.191587 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.191698 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.191921 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 13:56:26.192371 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.180830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.22.26.253:45111)
0927 13:56:26.180967 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.180972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.180988 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:56:26.181082 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.181087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.181145 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.181245 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.181856 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.182531 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.183719 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.184135 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.184264 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.187134 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.187140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.187144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.187183 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.189966 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.190585 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.190589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.190591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.190717 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.191031 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.191034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.191036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.191352 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.191456 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.191783 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.192056 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":42}
I20260927 13:56:26.196897 29328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: Bootstrap starting.
I20260927 13:56:26.199967 29328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:26.200973 29328 log.cc:824] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:26.203591 29328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: No bootstrap required, opened a new log
I20260927 13:56:26.210451 29328 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } }
I20260927 13:56:26.210759 29328 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:26.210857 29328 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f5a277ee88440eb96a393fe2489a5b, State: Initialized, Role: FOLLOWER
I20260927 13:56:26.211513 29328 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } }
I20260927 13:56:26.212322 29335 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } } }
I20260927 13:56:26.212651 29335 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:56:26.213232 29328 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: configured and running, proceeding with master startup.
W20260927 13:56:26.221655 29346 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:56:26.221796 29346 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:56:26.223799 29261 master_runner.cc:428] Detected that this master d9f5a277ee88440eb96a393fe2489a5b is joining an existing cluster
I20260927 13:56:26.223910 29261 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.26.253:45111
I20260927 13:56:26.243295 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.228584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.22.26.254:39559)
0927 13:56:26.228791 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.228799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.228821 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:56:26.228883 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.228891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.229008 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.229116 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.230803 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.231952 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.233839 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.234255 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.234384 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.236326 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.236331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.236335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.236366 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.239858 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.240681 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.240686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.240690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.240820 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.241248 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.241253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.241256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.241610 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.241772 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.242622 (+ 850us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.242907 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 13:56:26.244202 29352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.227486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51092)
0927 13:56:26.228111 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.228131 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.228274 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.229181 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.229199 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.230076 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:26.230584 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.230601 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.232092 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.232100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.233684 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.233696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.234447 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.234453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.234646 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.235360 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:26.235378 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:26.236133 (+ 755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:26.240003 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.240007 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.240011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.240565 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.240940 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.240946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.240953 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.241107 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.241842 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:26.241845 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:26.242046 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.242205 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.243817 (+ 1612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":1281,"threads_started":1}
I20260927 13:56:26.247289 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.228127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:45111 (local address 127.0.0.1:56826)
0927 13:56:26.231503 (+ 3376us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.231521 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.231637 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.232045 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.232051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.232747 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:26.233091 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.233104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.234567 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.234575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.236521 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.236536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.237723 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.237731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.237894 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.238587 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:26.238612 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:26.239626 (+ 1014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:26.244513 (+ 4887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.244517 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.244521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.245010 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.245403 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.245407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.245411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.245541 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.246211 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:26.246215 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:26.246420 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.246557 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.246803 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3253,"mutex_wait_us":4,"thread_start_us":3044,"threads_started":1}
I20260927 13:56:26.248127 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.228625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.22.26.253:45111)
0927 13:56:26.228767 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.228771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.231610 (+ 2839us) server_negotiation.cc:407] Connection header received
0927 13:56:26.231748 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.231753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.231815 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.231899 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.233250 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.234342 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.236667 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.237348 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.237476 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.239816 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.239822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.239825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.239861 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.244382 (+ 4521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.245127 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.245131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.245134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.245292 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.245641 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.245645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.245648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.246002 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.246149 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.247551 (+ 1402us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.247783 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for oryx/127.22.26.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.26.252@KRBTEST.COM" created.
Sep 27 13:56:26 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517383, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.254@KRBTEST.COM for oryx/127.22.26.253@KRBTEST.COM
I20260927 13:56:26.272343 29355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.250948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:45111 (local address 127.0.0.1:56842)
0927 13:56:26.254874 (+ 3926us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.254911 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.255105 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.255583 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.255588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.256161 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:26.256370 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.256379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.257831 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.257839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.259634 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.259652 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.260688 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.260693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.260939 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.261682 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:26.261721 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:26.264663 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:26.269068 (+ 4405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.269080 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.269088 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.269657 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.270141 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.270146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.270150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.270341 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.271146 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:26.271156 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:26.271373 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.271575 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.271908 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3700,"thread_start_us":3558,"threads_started":1}
I20260927 13:56:26.272781 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.250947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.22.26.253:45111)
0927 13:56:26.251148 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.251156 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.255026 (+ 3870us) server_negotiation.cc:407] Connection header received
0927 13:56:26.255308 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.255314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.255417 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.255521 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.256512 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.257684 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.259789 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.260502 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.260626 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.264867 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.264876 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.264882 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.264922 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.268906 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.269795 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.269803 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.269810 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.269978 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.270476 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.270483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.270490 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.270947 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.271155 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.271719 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.271970 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:56:26.274695 28799 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.26.253:45111: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b"
member_type: NON_VOTER
last_known_addr {
host: "127.22.26.253"
port: 45111
}
attrs {
promote: true
}
}
dest_uuid: "ae445534f86344d9bf1ef1b1e0328b91"
cas_config_opid_index: -1
I20260927 13:56:26.276031 28799 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } }
I20260927 13:56:26.280830 29303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:26.279909 29358 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d9f5a277ee88440eb96a393fe2489a5b (127.22.26.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } } }
I20260927 13:56:26.283912 29303 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 FOLLOWER]: Refusing update from remote peer ae445534f86344d9bf1ef1b1e0328b91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 13:56:26.284863 29359 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.26.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.252.keytab.
Entry for principal oryx/127.22.26.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.252.keytab.
I20260927 13:56:26.287264 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.252:39795
--webserver_interface=127.22.26.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.252.keytab
--principal=oryx/127.22.26.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:39559,127.22.26.253:45111,127.22.26.252:39795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:56:26.291075 29358 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.26.253:45111
I20260927 13:56:26.292850 29261 master.cc:537] Master@127.22.26.253:45111 shutting down...
I20260927 13:56:26.292625 29360 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } } }
I20260927 13:56:26.293023 29360 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:26.303308 29303 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d9f5a277ee88440eb96a393fe2489a5b (127.22.26.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } } }
I20260927 13:56:26.307896 29368 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } } }
I20260927 13:56:26.308478 29368 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: This master's current role is: LEARNER
I20260927 13:56:26.308832 29365 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } } }
I20260927 13:56:26.309274 29365 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: This master's current role is: LEARNER
I20260927 13:56:26.309252 29335 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae445534f86344d9bf1ef1b1e0328b91. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } } }
I20260927 13:56:26.309525 29335 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: This master's current role is: LEARNER
I20260927 13:56:26.323915 29261 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 LEARNER]: Raft consensus shutting down.
I20260927 13:56:26.324443 29261 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 LEARNER]: Raft consensus is shut down!
I20260927 13:56:26.324617 29261 tablet_replica.cc:330] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: stopping tablet replica
I20260927 13:56:26.342232 29261 master.cc:552] Master@127.22.26.253:45111 shutdown complete.
I20260927 13:56:26.342325 29261 master_runner.cc:305] Clearing existing system tablet
I20260927 13:56:26.342746 29261 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:26.342901 29261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:26.346025 29261 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 13:56:26.348140 29370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:26.348551 29261 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:26.348886 29261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:56:26.348989 29261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/wal
uuid: "d9f5a277ee88440eb96a393fe2489a5b"
format_stamp: "Formatted at 2026-09-27 13:56:26 on dist-test-slave-gkks"
server_key: "fedb42c0a1be1f85f33f6c8b95fcd94c"
server_key_iv: "6eacda647cc45b2148206236f17f78bd"
server_key_version: "encryptionkey@0"
I20260927 13:56:26.353964 29261 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:26.360733 29261 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 13:56:26.360898 29261 log.cc:1196] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 13:56:26.361485 29261 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: Deleting consensus metadata
I20260927 13:56:26.361744 29261 master_runner.cc:315] Copying system tablet from 127.22.26.254:39559
I20260927 13:56:26.367414 29261 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: tablet copy: Beginning tablet copy session from remote peer at address 127.22.26.254:39559
I20260927 13:56:26.383394 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.368201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.22.26.254:39559)
0927 13:56:26.368404 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.368411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.368944 (+ 533us) server_negotiation.cc:407] Connection header received
0927 13:56:26.369172 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.369179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.369264 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.369375 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.370565 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.371741 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.373745 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.374399 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.374532 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.376728 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.376735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.376741 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.376778 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.380265 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.381161 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.381167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.381173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.381319 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.381777 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.381783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.381786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.382126 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.382283 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.382776 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.383002 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:56:26.383590 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.368159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51104)
0927 13:56:26.368817 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.368836 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.368988 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.369432 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.369439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.369990 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:26.370313 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.370333 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.371899 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.371904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.373593 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.373609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.374550 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.374555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.374733 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.375460 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:26.375483 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:26.376529 (+ 1046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:26.380431 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.380436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.380440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.381026 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.381459 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.381464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.381468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.381655 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.382267 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:26.382272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:26.382470 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.382657 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.382959 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":387,"threads_started":1}
I20260927 13:56:26.384677 28820 tablet_copy_service.cc:140] P ae445534f86344d9bf1ef1b1e0328b91: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d9f5a277ee88440eb96a393fe2489a5b ({username='oryx', principal='oryx/127.22.26.253@KRBTEST.COM'} at 127.0.0.1:51104)
I20260927 13:56:26.385041 28820 tablet_copy_service.cc:161] P ae445534f86344d9bf1ef1b1e0328b91: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d9f5a277ee88440eb96a393fe2489a5b at {username='oryx', principal='oryx/127.22.26.253@KRBTEST.COM'} at 127.0.0.1:51104: session id = d9f5a277ee88440eb96a393fe2489a5b-00000000000000000000000000000000
I20260927 13:56:26.389465 28820 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 13:56:26.391666 29261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 13:56:26.398281 29261 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: tablet copy: Starting download of 0 data blocks...
I20260927 13:56:26.398566 29261 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: tablet copy: Starting download of 1 WAL segments...
I20260927 13:56:26.401273 29261 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 13:56:26.405730 28820 tablet_copy_service.cc:342] P ae445534f86344d9bf1ef1b1e0328b91: Request end of tablet copy session d9f5a277ee88440eb96a393fe2489a5b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.26.253@KRBTEST.COM'} at 127.0.0.1:51104
I20260927 13:56:26.405932 28820 tablet_copy_service.cc:433] P ae445534f86344d9bf1ef1b1e0328b91: ending tablet copy session d9f5a277ee88440eb96a393fe2489a5b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d9f5a277ee88440eb96a393fe2489a5b
I20260927 13:56:26.417805 29261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:26.420369 29377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:26.424412 29378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:26.431325 29380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:26.436443 29261 server_base.cc:1034] running on GCE node
I20260927 13:56:26.436918 29261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:26.437546 29261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:26.438771 29261 hybrid_clock.cc:648] HybridClock initialized: now 1790517386438727 us; error 74 us; skew 500 ppm
I20260927 13:56:26.438985 29261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:26 dist-test-slave-gkks krb5kdc[28755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517386, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:26.441207 29261 init.cc:377] Logged in from keytab as oryx/127.22.26.253@KRBTEST.COM (short username oryx)
I20260927 13:56:26.445127 29261 webserver.cc:460] Webserver started at http://127.22.26.253:34711/ using document root <none> and password file <none>
I20260927 13:56:26.445506 29261 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:26.445585 29261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:26.448769 29261 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:56:26.450989 29387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:26.451498 29261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:56:26.451671 29261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/wal
uuid: "d9f5a277ee88440eb96a393fe2489a5b"
format_stamp: "Formatted at 2026-09-27 13:56:26 on dist-test-slave-gkks"
server_key: "fedb42c0a1be1f85f33f6c8b95fcd94c"
server_key_iv: "6eacda647cc45b2148206236f17f78bd"
server_key_version: "encryptionkey@0"
I20260927 13:56:26.451903 29261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:26.488369 29261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:26.492794 29261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:26.532181 29261 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.253:45111
I20260927 13:56:26.532143 29438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.253:45111 every 8 connection(s)
I20260927 13:56:26.533288 29261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-1/data/info.pb
I20260927 13:56:26.538769 29439 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:56:26.543009 29439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: Bootstrap starting.
I20260927 13:56:26.558499 29439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:56:26.559010 29439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: Bootstrap complete.
I20260927 13:56:26.560068 29439 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } }
I20260927 13:56:26.560324 29439 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d9f5a277ee88440eb96a393fe2489a5b, State: Initialized, Role: LEARNER
I20260927 13:56:26.560577 29439 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } }
I20260927 13:56:26.561417 29442 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } } }
I20260927 13:56:26.561797 29442 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: This master's current role is: LEARNER
I20260927 13:56:26.561909 29439 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:26.570173 29453 catalog_manager.cc:1259] Loaded cluster ID: 592198b8ff06425389edd11acf820348
I20260927 13:56:26.570731 29453 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: loading cluster ID for follower catalog manager: success
I20260927 13:56:26.574347 29453 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: acquiring CA information for follower catalog manager: success
I20260927 13:56:26.576776 29453 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:56:26.647833 29256 heartbeater.cc:503] Master 127.22.26.254:39559 was elected leader, sending a full tablet report...
W20260927 13:56:26.677253 29362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:26.677554 29362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:26.677717 29362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:26.685245 29362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:26.685377 29362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:26.685475 29362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:26.685575 29362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:26.685648 29362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:56:26.695217 29362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:39559,127.22.26.253:45111,127.22.26.252:39795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.252.keytab
--principal=oryx/127.22.26.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.252:39795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.26.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:26.696439 29362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:26.697644 29362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:26.708863 29461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:26.708904 29459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:26.708819 29458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:26.710083 29362 server_base.cc:1034] running on GCE node
I20260927 13:56:26.710790 29362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:26.712236 29362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:26.713456 29362 hybrid_clock.cc:648] HybridClock initialized: now 1790517386713418 us; error 54 us; skew 500 ppm
I20260927 13:56:26.713783 29362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:26 dist-test-slave-gkks krb5kdc[28755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517386, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:26.716745 29362 init.cc:377] Logged in from keytab as oryx/127.22.26.252@KRBTEST.COM (short username oryx)
I20260927 13:56:26.718092 29362 webserver.cc:460] Webserver started at http://127.22.26.252:41677/ using document root <none> and password file <none>
I20260927 13:56:26.718722 29362 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:26.718863 29362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:26.719226 29362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:26.721993 29362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/data/instance:
uuid: "bc1e0ccd48fb468e80426dec238012ff"
format_stamp: "Formatted at 2026-09-27 13:56:26 on dist-test-slave-gkks"
server_key: "e225fef88a206e1cbca64ad221315821"
server_key_iv: "0fa916e692f5982026c73320c56b4d78"
server_key_version: "encryptionkey@0"
I20260927 13:56:26.722729 29362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/wal/instance:
uuid: "bc1e0ccd48fb468e80426dec238012ff"
format_stamp: "Formatted at 2026-09-27 13:56:26 on dist-test-slave-gkks"
server_key: "e225fef88a206e1cbca64ad221315821"
server_key_iv: "0fa916e692f5982026c73320c56b4d78"
server_key_version: "encryptionkey@0"
I20260927 13:56:26.727443 29362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.005s
I20260927 13:56:26.730623 29468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:26.731884 29362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:26.732052 29362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/wal
uuid: "bc1e0ccd48fb468e80426dec238012ff"
format_stamp: "Formatted at 2026-09-27 13:56:26 on dist-test-slave-gkks"
server_key: "e225fef88a206e1cbca64ad221315821"
server_key_iv: "0fa916e692f5982026c73320c56b4d78"
server_key_version: "encryptionkey@0"
I20260927 13:56:26.732239 29362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:26.745494 29362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:26.749243 29362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:26.749511 29362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:26.767782 29362 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.252:39795
I20260927 13:56:26.767805 29519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.252:39795 every 8 connection(s)
I20260927 13:56:26.769102 29362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/data/info.pb
I20260927 13:56:26.772847 29520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:26.776381 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 29362
I20260927 13:56:26.780519 29520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:56:26 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.26.252@KRBTEST.COM
Sep 27 13:56:26 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517386, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.252@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
I20260927 13:56:26.798209 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.783419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.22.26.254:39559)
0927 13:56:26.783579 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.783583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.784139 (+ 556us) server_negotiation.cc:407] Connection header received
0927 13:56:26.784342 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.784348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.784419 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.784519 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.785717 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.786465 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.788168 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.788574 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.788669 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.792662 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.792667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.792670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.792708 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.795349 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.796074 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.796078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.796080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.796209 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.796607 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.796611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.796613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.796877 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.796983 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.797620 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.797850 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:56:26.799259 29524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.783375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51112)
0927 13:56:26.784016 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.784052 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.784211 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.784571 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.784577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.785257 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:26.785566 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.785579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.786614 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.786618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.787999 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.788037 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.788800 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.788808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.789010 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.789605 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:26.789636 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:26.792465 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:26.795461 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.795470 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.795483 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.795933 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.796336 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.796339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.796341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.796465 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.797004 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:26.797012 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:26.797208 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.797497 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.797998 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":126,"thread_start_us":347,"threads_started":1}
I20260927 13:56:26.799294 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.778537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.22.26.252:39795)
0927 13:56:26.779304 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.779316 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.779365 (+ 49us) server_negotiation.cc:407] Connection header received
0927 13:56:26.781044 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.781064 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.781455 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.781835 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.783331 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.784995 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.786465 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.787248 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.787370 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.792415 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.792427 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.792441 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.792476 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.795445 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.796024 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.796031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.796034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.796174 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.796558 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.796563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.796566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.797072 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.797397 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.797580 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.797897 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":578,"thread_start_us":329,"threads_started":1}
I20260927 13:56:26.800771 29520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:56:26 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517386, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.252@KRBTEST.COM for oryx/127.22.26.253@KRBTEST.COM
I20260927 13:56:26.813215 29524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.801543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:45111 (local address 127.0.0.1:56850)
0927 13:56:26.801717 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.801748 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.801859 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.802375 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.802379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.802697 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:26.802825 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.802833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.804136 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.804141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.805051 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.805062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.805782 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.805788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.805969 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.806403 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:26.806419 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:26.808445 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:26.811403 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.811407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.811409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.811745 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.812042 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.812045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.812047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.812158 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.812602 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:26.812604 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:26.812712 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.812815 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.812996 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 13:56:26.813359 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.801574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.22.26.253:45111)
0927 13:56:26.802062 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.802066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.802080 (+ 14us) server_negotiation.cc:407] Connection header received
0927 13:56:26.802149 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.802154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.802239 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.802355 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.802965 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.803994 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.805186 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.805619 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.805762 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.808635 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.808639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.808642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.808673 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.811294 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.811840 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.811843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.811846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.811953 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.812266 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.812269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.812271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.812511 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.812624 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.812895 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.813047 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":235,"threads_started":1}
I20260927 13:56:26.814344 29520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:56:26 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517386, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.252@KRBTEST.COM for oryx/127.22.26.252@KRBTEST.COM
I20260927 13:56:26.825232 29524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.814920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:39795 (local address 127.0.0.1:35310)
0927 13:56:26.815069 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.815109 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.815196 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.815442 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.815445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.815733 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:26.815853 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.815860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.816754 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.816758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.817581 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.817592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.818189 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.818193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.818342 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.818820 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:26.818835 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:26.820727 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:26.823384 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.823387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.823390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.823738 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.824026 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.824029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.824031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.824165 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.824567 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:26.824570 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:26.824666 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.824797 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.824958 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 13:56:26.825282 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.814983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.22.26.252:39795)
0927 13:56:26.815157 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.815162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.815176 (+ 14us) server_negotiation.cc:407] Connection header received
0927 13:56:26.815284 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.815288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.815335 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.815409 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.815962 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.816590 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.817700 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.818070 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.818167 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.820882 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.820887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.820890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.820917 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.823271 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.823829 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.823833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.823835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.823940 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.824250 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.824253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.824254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.824476 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.824575 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.824879 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.825055 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":52}
I20260927 13:56:26.829877 29520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: Bootstrap starting.
I20260927 13:56:26.832904 29520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:26.833894 29520 log.cc:824] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:26.836781 29520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: No bootstrap required, opened a new log
I20260927 13:56:26.843667 29520 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } }
I20260927 13:56:26.843990 29520 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:26.844113 29520 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc1e0ccd48fb468e80426dec238012ff, State: Initialized, Role: FOLLOWER
I20260927 13:56:26.844692 29520 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } }
I20260927 13:56:26.845489 29527 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } } }
I20260927 13:56:26.845824 29527 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:56:26.846480 29520 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:26.854061 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 28861
W20260927 13:56:26.856835 29538 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:56:26.857569 29538 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:56:26.859323 29362 master_runner.cc:428] Detected that this master bc1e0ccd48fb468e80426dec238012ff is joining an existing cluster
I20260927 13:56:26.859429 29362 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.26.252:39795
Sep 27 13:56:26 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.26.193@KRBTEST.COM
I20260927 13:56:26.880314 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.855409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.22.26.193:43167)
0927 13:56:26.855963 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.855969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.855990 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:56:26.856042 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.856048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.856293 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.857813 (+ 1520us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.858119 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.859557 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.862170 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.863564 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.864571 (+ 1007us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.870221 (+ 5650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.870255 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.870276 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.870325 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.875526 (+ 5201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.876658 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.876671 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.876677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.876898 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.877524 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.877531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.877537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.878016 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.879467 (+ 1451us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.879622 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.879887 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":267,"threads_started":1}
I20260927 13:56:26.881955 28984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:26.901787 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.881396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.22.26.253:45111)
0927 13:56:26.881594 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.881602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.882162 (+ 560us) server_negotiation.cc:407] Connection header received
0927 13:56:26.882493 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.882501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.882619 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.882706 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.884580 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.886021 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.888788 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.889676 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.889795 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.893094 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.893104 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.893111 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.893162 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.897674 (+ 4512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.898691 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.898700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.898706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.898893 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.899533 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.899541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.899547 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.899977 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.900139 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.900951 (+ 812us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.901277 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:56:26.903028 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.881151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:45111 (local address 127.0.0.1:56860)
0927 13:56:26.882056 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.882076 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.882297 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.882973 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.882980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.883904 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:26.884361 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.884378 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.886265 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.886273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.888566 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.888586 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.890164 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.890173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.890426 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.891286 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:26.891318 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:26.892827 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:26.897895 (+ 5068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.897903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.897909 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.898527 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.899069 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.899076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.899135 (+ 59us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.899373 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.900255 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:26.900262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:26.900570 (+ 308us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.902043 (+ 1473us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.902438 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":730,"threads_started":1}
I20260927 13:56:26.911268 29527 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:56:26.911545 29527 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } }
I20260927 13:56:26.913265 29527 leader_election.cc:288] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae445534f86344d9bf1ef1b1e0328b91 (127.22.26.254:39559), d9f5a277ee88440eb96a393fe2489a5b (127.22.26.253:45111)
I20260927 13:56:26.913652 29541 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } } }
I20260927 13:56:26.913846 29414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc1e0ccd48fb468e80426dec238012ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f5a277ee88440eb96a393fe2489a5b" is_pre_election: true
I20260927 13:56:26.914057 29541 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:26.914201 29414 raft_consensus.cc:1799] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 LEARNER]: Handling vote request from an unknown peer bc1e0ccd48fb468e80426dec238012ff
I20260927 13:56:26.914268 28810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc1e0ccd48fb468e80426dec238012ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae445534f86344d9bf1ef1b1e0328b91" is_pre_election: true
I20260927 13:56:26.914594 29414 raft_consensus.cc:2396] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate bc1e0ccd48fb468e80426dec238012ff for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260927 13:56:26.915397 29469 leader_election.cc:302] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc1e0ccd48fb468e80426dec238012ff; no voters: ae445534f86344d9bf1ef1b1e0328b91, d9f5a277ee88440eb96a393fe2489a5b
I20260927 13:56:26.916296 29527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:26.921437 29527 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 13:56:26.922494 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.902715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.22.26.253:45111)
0927 13:56:26.904074 (+ 1359us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.904080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.911706 (+ 7626us) server_negotiation.cc:407] Connection header received
0927 13:56:26.912234 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.912239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.912326 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.912407 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:26.914336 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.915393 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.919853 (+ 4460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.921107 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.921228 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.921446 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.922073 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.922234 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1243}
I20260927 13:56:26.923182 29551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.902322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:45111 (local address 127.0.0.1:56872)
0927 13:56:26.911584 (+ 9262us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.911610 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.912058 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.912683 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.912691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.912724 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:26.913040 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.913053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.915648 (+ 2595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.915656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.918292 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.918308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.921617 (+ 3309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.921623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.921829 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.921915 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.922652 (+ 737us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.922804 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9090,"thread_start_us":9401,"threads_started":1}
I20260927 13:56:26.924216 29442 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae445534f86344d9bf1ef1b1e0328b91. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } } }
I20260927 13:56:26.924453 29442 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: This master's current role is: LEARNER
I20260927 13:56:26.925104 29546 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91: attempting to promote NON_VOTER d9f5a277ee88440eb96a393fe2489a5b to VOTER
I20260927 13:56:26.926261 29546 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } }
I20260927 13:56:26.928121 29414 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 LEARNER]: Refusing update from remote peer ae445534f86344d9bf1ef1b1e0328b91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 13:56:26.928764 29541 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 13:56:26.929704 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.882413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51116)
0927 13:56:26.905144 (+ 22731us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.905174 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.905393 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.906013 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.906026 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.906692 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:26.906945 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.906960 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.909003 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.909016 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.911100 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.911118 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.912332 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.912337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.912493 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.913132 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:26.913153 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:26.920981 (+ 7828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:26.925370 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.925375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.925383 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.926105 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.926687 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.926692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.926696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.926863 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.927674 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:26.927679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:26.927840 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.928797 (+ 957us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.929208 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22594,"thread_start_us":6801,"threads_started":1}
I20260927 13:56:26.930519 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.880641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.22.26.254:39559)
0927 13:56:26.880827 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.880834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.905277 (+ 24443us) server_negotiation.cc:407] Connection header received
0927 13:56:26.905557 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.905566 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.905669 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.905755 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.907204 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.908729 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.911307 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.911969 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.912077 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.921203 (+ 9126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.921212 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.921220 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.921266 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.925135 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.926244 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.926253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.926257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.926472 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.926991 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.926997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.927000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.927429 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.927590 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.928193 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.928540 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:56:26.934187 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.881947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:39795 (local address 127.0.0.1:35322)
0927 13:56:26.889110 (+ 7163us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.889128 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.889491 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.889931 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.889935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.890565 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:26.890894 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.890906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.892439 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.892444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.894023 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.894036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.896819 (+ 2783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.896827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.897078 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.899539 (+ 2461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:26.899570 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:26.900955 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:26.931099 (+ 30144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.931104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.931108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.931626 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.932079 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.932084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.932088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.932256 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.933004 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:26.933008 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:26.933164 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.933287 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.933649 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7025,"thread_start_us":37624,"threads_started":1}
I20260927 13:56:26.935297 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.881683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.22.26.252:39795)
0927 13:56:26.881879 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.881884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.889205 (+ 7321us) server_negotiation.cc:407] Connection header received
0927 13:56:26.889614 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.889619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.889683 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.889743 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.891043 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.892206 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.895340 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.896251 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.896364 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.918624 (+ 22260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.918638 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.918646 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.918691 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.930509 (+ 11818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.931798 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.931805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.931810 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.931953 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.932370 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.932375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.932379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.932802 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.932954 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.934629 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.934887 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:56:26.938387 29546 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d9f5a277ee88440eb96a393fe2489a5b (127.22.26.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } }
I20260927 13:56:26.942135 29546 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } }
I20260927 13:56:26.942363 29546 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:26.942570 29546 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } }
I20260927 13:56:26.942780 29546 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:26.943553 29414 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d9f5a277ee88440eb96a393fe2489a5b (127.22.26.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } }
Sep 27 13:56:26 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517383, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.254@KRBTEST.COM for oryx/127.22.26.252@KRBTEST.COM
I20260927 13:56:26.947803 29442 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } }
I20260927 13:56:26.948014 29442 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:56:26.954578 29551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.936073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:39795 (local address 127.0.0.1:35338)
0927 13:56:26.937625 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:26.937648 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:26.937824 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:26.938443 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:26.938448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:26.939069 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:26.939282 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.939293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.940692 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.940698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.942163 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:26.942187 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.943638 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.943644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:26.943824 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.944512 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:26.944534 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:26.947723 (+ 3189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:26.951813 (+ 4090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.951819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.951823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.952374 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.952767 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:26.952774 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:26.952778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:26.952942 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:26.953577 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:26.953582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:26.953784 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 13:56:26.953944 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 13:56:26.954227 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1419}
I20260927 13:56:26.954600 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:26.936395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.22.26.252:39795)
0927 13:56:26.937205 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:26.937213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:26.937718 (+ 505us) server_negotiation.cc:407] Connection header received
0927 13:56:26.938006 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:26.938012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:26.938132 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:26.938212 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:26.939447 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.940525 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.942328 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:26.943121 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:26.943226 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:26.947935 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:26.947941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:26.947946 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:26.947997 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:26.951673 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.952505 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.952510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.952514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.952646 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:26.953054 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:26.953059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:26.953062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:26.953459 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:26.953593 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:26.954044 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 13:56:26.954270 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653}
I20260927 13:56:26.956151 28799 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.26.252:39795: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff"
member_type: NON_VOTER
last_known_addr {
host: "127.22.26.252"
port: 39795
}
attrs {
promote: true
}
}
dest_uuid: "ae445534f86344d9bf1ef1b1e0328b91"
cas_config_opid_index: 6
I20260927 13:56:26.956828 28799 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: NON_VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: true } }
I20260927 13:56:26.959398 29495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 FOLLOWER]: Refusing update from remote peer ae445534f86344d9bf1ef1b1e0328b91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 13:56:26.960266 29414 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 FOLLOWER]: Refusing update from remote peer ae445534f86344d9bf1ef1b1e0328b91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 13:56:26.960237 29546 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: NON_VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:26.960852 29541 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 13:56:26.968379 29546 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bc1e0ccd48fb468e80426dec238012ff (127.22.26.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: NON_VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: true } } }
I20260927 13:56:26.972535 29495 raft_consensus.cc:858] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } }
W20260927 13:56:26.972771 29495 raft_consensus.cc:1565] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } }
I20260927 13:56:26.972929 29414 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bc1e0ccd48fb468e80426dec238012ff (127.22.26.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: NON_VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: true } } }
I20260927 13:56:26.973568 29546 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.26.252:39795
I20260927 13:56:26.976186 29362 master.cc:537] Master@127.22.26.252:39795 shutting down...
I20260927 13:56:26.977293 29541 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: NON_VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: true } } }
I20260927 13:56:26.977572 29541 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:26.977789 29541 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: NON_VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: true } } }
I20260927 13:56:26.977993 29541 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:26.979449 29442 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: NON_VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: true } } }
I20260927 13:56:26.979758 29442 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:56:27.021492 29495 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d9f5a277ee88440eb96a393fe2489a5b (127.22.26.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } } }
I20260927 13:56:27.039486 29560 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } } }
I20260927 13:56:27.039844 29560 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 13:56:27.040081 29527 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae445534f86344d9bf1ef1b1e0328b91. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } } }
I20260927 13:56:27.040361 29527 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 13:56:27.040122 29557 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: NON_VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: true } } }
I20260927 13:56:27.040493 29557 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 13:56:27.042498 29546 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: NON_VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: true } } }
I20260927 13:56:27.042781 29546 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: This master's current role is: LEADER
W20260927 13:56:27.043591 28787 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 13:56:27.044179 29362 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 13:56:27.044653 29362 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 13:56:27.044847 29362 tablet_replica.cc:330] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: stopping tablet replica
I20260927 13:56:27.053117 29362 master.cc:552] Master@127.22.26.252:39795 shutdown complete.
I20260927 13:56:27.053207 29362 master_runner.cc:305] Clearing existing system tablet
I20260927 13:56:27.053802 29362 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:27.053903 29362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:27.062160 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 28861
W20260927 13:56:27.062356 28787 consensus_peers.cc:487] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 -> Peer bc1e0ccd48fb468e80426dec238012ff (127.22.26.252:39795): Couldn't send request to peer bc1e0ccd48fb468e80426dec238012ff. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260927 13:56:27.063117 29362 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.005s
I20260927 13:56:27.065927 29563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:27.066447 29362 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:27.066809 29362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 13:56:27.066942 29362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/wal
uuid: "bc1e0ccd48fb468e80426dec238012ff"
format_stamp: "Formatted at 2026-09-27 13:56:26 on dist-test-slave-gkks"
server_key: "e225fef88a206e1cbca64ad221315821"
server_key_iv: "0fa916e692f5982026c73320c56b4d78"
server_key_version: "encryptionkey@0"
I20260927 13:56:27.070305 29362 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:27.074887 29362 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 13:56:27.075048 29362 log.cc:1196] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 13:56:27.075555 29362 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: Deleting consensus metadata
I20260927 13:56:27.075799 29362 master_runner.cc:315] Copying system tablet from 127.22.26.254:39559
I20260927 13:56:27.078253 29362 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: tablet copy: Beginning tablet copy session from remote peer at address 127.22.26.254:39559
I20260927 13:56:27.085917 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:43167
--local_ip_for_outbound_sockets=127.22.26.193
--tserver_master_addrs=127.22.26.254:39559
--webserver_port=46135
--webserver_interface=127.22.26.193
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab
--principal=oryx/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.26.254:39559,127.22.26.253:45111
--tserver_master_addrs=127.22.26.254:39559,127.22.26.253:45111,127.22.26.252:39795 with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:56:27.093468 29569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:27.078940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51128)
0927 13:56:27.079633 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:27.079654 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:27.079811 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:27.080239 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:27.080245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:27.080756 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:27.080979 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:27.080989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.081995 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.082000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:27.082925 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:27.082936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.083783 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.083790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:27.083977 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:27.084616 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:27.084634 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:27.086092 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:27.090552 (+ 4460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:27.090558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:27.090562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:27.091056 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:27.091502 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:27.091507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:27.091511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:27.091675 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:27.092338 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:27.092343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:27.092536 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 13:56:27.092699 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 13:56:27.093006 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":461,"threads_started":1}
I20260927 13:56:27.094338 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:27.078976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.22.26.254:39559)
0927 13:56:27.079231 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:27.079241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:27.079745 (+ 504us) server_negotiation.cc:407] Connection header received
0927 13:56:27.079957 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:27.079962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:27.080044 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:27.080155 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:27.081122 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.081843 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.083074 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.083586 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.083745 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:27.086323 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:27.086332 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:27.086340 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:27.086399 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:27.090404 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:27.091215 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:27.091223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:27.091230 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:27.091381 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:27.091794 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:27.091801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:27.091808 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:27.092206 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:27.092366 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:27.093759 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0927 13:56:27.093990 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:56:27.094815 28820 tablet_copy_service.cc:140] P ae445534f86344d9bf1ef1b1e0328b91: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bc1e0ccd48fb468e80426dec238012ff ({username='oryx', principal='oryx/127.22.26.252@KRBTEST.COM'} at 127.0.0.1:51128)
I20260927 13:56:27.095001 28820 tablet_copy_service.cc:161] P ae445534f86344d9bf1ef1b1e0328b91: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bc1e0ccd48fb468e80426dec238012ff at {username='oryx', principal='oryx/127.22.26.252@KRBTEST.COM'} at 127.0.0.1:51128: session id = bc1e0ccd48fb468e80426dec238012ff-00000000000000000000000000000000
I20260927 13:56:27.099213 28820 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 13:56:27.101127 29362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 13:56:27.107972 29362 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: tablet copy: Starting download of 0 data blocks...
I20260927 13:56:27.108273 29362 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: tablet copy: Starting download of 1 WAL segments...
I20260927 13:56:27.110920 29362 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 13:56:27.115404 28820 tablet_copy_service.cc:342] P ae445534f86344d9bf1ef1b1e0328b91: Request end of tablet copy session bc1e0ccd48fb468e80426dec238012ff-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.26.252@KRBTEST.COM'} at 127.0.0.1:51128
I20260927 13:56:27.115536 28820 tablet_copy_service.cc:433] P ae445534f86344d9bf1ef1b1e0328b91: ending tablet copy session bc1e0ccd48fb468e80426dec238012ff-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bc1e0ccd48fb468e80426dec238012ff
I20260927 13:56:27.128338 29362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:27.131011 29571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:27.136531 29572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:27.143440 29574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:27.148483 29362 server_base.cc:1034] running on GCE node
I20260927 13:56:27.148870 29362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:27.149541 29362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:27.150756 29362 hybrid_clock.cc:648] HybridClock initialized: now 1790517387150695 us; error 52 us; skew 500 ppm
I20260927 13:56:27.151014 29362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:27 dist-test-slave-gkks krb5kdc[28755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517387, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:27.153285 29362 init.cc:377] Logged in from keytab as oryx/127.22.26.252@KRBTEST.COM (short username oryx)
I20260927 13:56:27.160236 29362 webserver.cc:460] Webserver started at http://127.22.26.252:43389/ using document root <none> and password file <none>
I20260927 13:56:27.160657 29362 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:27.160789 29362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:27.164088 29362 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:56:27.166080 29581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:27.166559 29362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 13:56:27.166733 29362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/wal
uuid: "bc1e0ccd48fb468e80426dec238012ff"
format_stamp: "Formatted at 2026-09-27 13:56:26 on dist-test-slave-gkks"
server_key: "e225fef88a206e1cbca64ad221315821"
server_key_iv: "0fa916e692f5982026c73320c56b4d78"
server_key_version: "encryptionkey@0"
I20260927 13:56:27.166919 29362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:27.178951 29362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:27.183531 29362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:27.206301 29362 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.252:39795
I20260927 13:56:27.206530 29632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.252:39795 every 8 connection(s)
I20260927 13:56:27.207335 29362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/master-2/data/info.pb
I20260927 13:56:27.212822 29633 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:56:27.216851 29633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: Bootstrap starting.
I20260927 13:56:27.231987 29633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:56:27.232511 29633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: Bootstrap complete.
I20260927 13:56:27.233572 29633 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: NON_VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: true } }
I20260927 13:56:27.233867 29633 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bc1e0ccd48fb468e80426dec238012ff, State: Initialized, Role: LEARNER
I20260927 13:56:27.234154 29633 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: NON_VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: true } }
I20260927 13:56:27.235335 29633 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:27.235134 29636 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: NON_VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: true } } }
I20260927 13:56:27.235742 29636 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: This master's current role is: LEARNER
I20260927 13:56:27.243831 29647 catalog_manager.cc:1259] Loaded cluster ID: 592198b8ff06425389edd11acf820348
I20260927 13:56:27.243930 29647 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: loading cluster ID for follower catalog manager: success
I20260927 13:56:27.246582 29647 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: acquiring CA information for follower catalog manager: success
I20260927 13:56:27.248430 29647 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 13:56:27.499547 29570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:27.499807 29570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:27.499953 29570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:27.507316 29570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:27.507433 29570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:27.507601 29570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:56:27.517212 29570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab
--principal=oryx/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:43167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=46135
--tserver_master_addrs=127.22.26.254:39559,127.22.26.253:45111,127.22.26.252:39795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:27.518469 29570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:27.519838 29570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:27.532476 29652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:27.532446 29653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:27.533226 29655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:27.534438 29570 server_base.cc:1034] running on GCE node
I20260927 13:56:27.535030 29570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:27.536063 29570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:27.537282 29570 hybrid_clock.cc:648] HybridClock initialized: now 1790517387537247 us; error 52 us; skew 500 ppm
I20260927 13:56:27.537566 29570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:27 dist-test-slave-gkks krb5kdc[28755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517387, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:27.540923 29570 init.cc:377] Logged in from keytab as oryx/127.22.26.193@KRBTEST.COM (short username oryx)
I20260927 13:56:27.542358 29570 webserver.cc:460] Webserver started at http://127.22.26.193:46135/ using document root <none> and password file <none>
I20260927 13:56:27.543210 29570 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:27.543359 29570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:27.548698 29570 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:56:27.551786 29663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:27.553241 29570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:27.553416 29570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "88dc53435f9d4db09dc2b185d4505baf"
format_stamp: "Formatted at 2026-09-27 13:56:24 on dist-test-slave-gkks"
server_key: "d5f6ed738b8dd1a12eaed04f33015db2"
server_key_iv: "00b69140478730c41792e492f8bf13b0"
server_key_version: "encryptionkey@0"
I20260927 13:56:27.554442 29570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:27.585698 29662 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: NON_VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: true } } }
I20260927 13:56:27.586007 29662 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:27.586385 29570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:27.593341 29570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:27.593735 29570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:27.597797 29570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:27.599522 29570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:27.599617 29570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:27.599743 29570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:27.599825 29570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:27.603041 29672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:27.586121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.22.26.252:39795)
0927 13:56:27.597611 (+ 11490us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:27.597616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:27.597633 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:56:27.597699 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:27.597703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:27.597770 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:27.597872 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:27.598347 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.599564 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.601192 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.601989 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.602140 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:27.602396 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:27.602565 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 13:56:27.602705 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11374,"thread_start_us":11267,"threads_started":1}
I20260927 13:56:27.603116 29670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:27.586025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:39795 (local address 127.0.0.1:35342)
0927 13:56:27.596930 (+ 10905us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:27.596962 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:27.597171 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:27.597939 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:27.597944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:27.597975 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:27.598188 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:27.598200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.599775 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.599779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:27.601029 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:27.601043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.602181 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.602187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:27.602381 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:27.602416 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:56:27.602543 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 13:56:27.602779 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10738,"thread_start_us":10656,"threads_started":1}
I20260927 13:56:27.604161 29608 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260927 13:56:27.604646 29636 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae445534f86344d9bf1ef1b1e0328b91. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: NON_VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: true } } }
I20260927 13:56:27.604975 29636 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: This master's current role is: LEARNER
I20260927 13:56:27.605373 29662 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91: attempting to promote NON_VOTER bc1e0ccd48fb468e80426dec238012ff to VOTER
I20260927 13:56:27.606312 29662 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } }
I20260927 13:56:27.608528 29608 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 LEARNER]: Refusing update from remote peer ae445534f86344d9bf1ef1b1e0328b91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 13:56:27.608829 29414 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 FOLLOWER]: Refusing update from remote peer ae445534f86344d9bf1ef1b1e0328b91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 13:56:27.609191 29668 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 13:56:27.609903 29668 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 13:56:27.612350 29669 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bc1e0ccd48fb468e80426dec238012ff (127.22.26.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } } }
I20260927 13:56:27.613445 29414 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bc1e0ccd48fb468e80426dec238012ff (127.22.26.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } } }
I20260927 13:56:27.613473 29608 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bc1e0ccd48fb468e80426dec238012ff (127.22.26.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } } }
I20260927 13:56:27.617645 29636 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } } }
I20260927 13:56:27.617894 29636 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:56:27.619103 29693 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } } }
I20260927 13:56:27.619467 29693 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:56:27.620359 29662 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } } }
I20260927 13:56:27.620656 29662 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:27.621064 29662 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } } }
I20260927 13:56:27.621261 29662 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:27.621544 29669 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae445534f86344d9bf1ef1b1e0328b91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } } }
I20260927 13:56:27.621838 29669 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:27.646183 29570 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:43167
I20260927 13:56:27.646263 29782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:43167 every 8 connection(s)
I20260927 13:56:27.647730 29570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:56:27.654090 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 29570
I20260927 13:56:27.654448 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 28995
Sep 27 13:56:27 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.26.194@KRBTEST.COM
Sep 27 13:56:27 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517387, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.253@KRBTEST.COM
Sep 27 13:56:27 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517387, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.252@KRBTEST.COM
Sep 27 13:56:27 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517387, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
I20260927 13:56:27.707293 29787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:27.651168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:38649 (local address 127.22.26.253:45111)
0927 13:56:27.676154 (+ 24986us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:27.676163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:27.684078 (+ 7915us) server_negotiation.cc:407] Connection header received
0927 13:56:27.684674 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:27.684683 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:27.684808 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:27.684915 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:27.690132 (+ 5217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.691360 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.694001 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.694653 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.694769 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:27.699688 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:27.699693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:27.699696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:27.699729 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:27.703336 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:27.704312 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:27.704318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:27.704323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:27.704482 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:27.705074 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:27.705079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:27.705083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:27.705435 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:27.705582 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:27.706610 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0927 13:56:27.706851 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24761,"thread_start_us":29397,"threads_started":1}
I20260927 13:56:27.708195 29672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:27.652054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:55537 (local address 127.22.26.252:39795)
0927 13:56:27.652307 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:27.652316 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:27.673665 (+ 21349us) server_negotiation.cc:407] Connection header received
0927 13:56:27.688922 (+ 15257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:27.688931 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:27.689042 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:27.689144 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:27.691342 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.692380 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.694449 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.695205 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.695314 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:27.701042 (+ 5728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:27.701050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:27.701055 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:27.701099 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:27.704615 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:27.705869 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:27.705874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:27.705879 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:27.706052 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:27.706612 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:27.706616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:27.706619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:27.706942 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:27.707060 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:27.707607 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 13:56:27.707837 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 13:56:27.708810 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:27.655982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.22.26.194:44945)
0927 13:56:27.674276 (+ 18294us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:27.674288 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:27.677095 (+ 2807us) server_negotiation.cc:407] Connection header received
0927 13:56:27.677206 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:27.677219 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:27.677543 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:27.677737 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:27.684151 (+ 6414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.686291 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.691401 (+ 5110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.692251 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.692343 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:27.700259 (+ 7916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:27.700273 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:27.700289 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:27.700332 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:27.704764 (+ 4432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:27.705672 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:27.705683 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:27.705687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:27.705850 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:27.706358 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:27.706363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:27.706366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:27.706777 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:27.707072 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:27.708213 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0927 13:56:27.708484 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18042,"thread_start_us":18304,"threads_started":1}
I20260927 13:56:27.709146 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:27.651513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:42103 (local address 127.22.26.254:39559)
0927 13:56:27.686401 (+ 34888us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:27.686411 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:27.689835 (+ 3424us) server_negotiation.cc:407] Connection header received
0927 13:56:27.690016 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:27.690021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:27.690120 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:27.690226 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:27.691656 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.692891 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.694854 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.695495 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.695620 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:27.702110 (+ 6490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:27.702119 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:27.702123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:27.702174 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:27.705885 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:27.706685 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:27.706690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:27.706693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:27.706842 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:27.707354 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:27.707359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:27.707361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:27.707716 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:27.707848 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:27.708582 (+ 734us) server_negotiation.cc:299] Negotiation successful
0927 13:56:27.708791 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34720,"thread_start_us":38459,"threads_started":1}
I20260927 13:56:27.710060 29790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:27.651891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:45111 (local address 127.22.26.193:38649)
0927 13:56:27.683939 (+ 32048us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:27.683966 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:27.684481 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:27.686647 (+ 2166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:27.686664 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:27.687941 (+ 1277us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:27.689087 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:27.689119 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.691605 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.691609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:27.693836 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:27.693850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.695157 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.695164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:27.695427 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:27.696166 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:27.696195 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:27.699470 (+ 3275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:27.703508 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:27.703519 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:27.703541 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:27.704165 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:27.704648 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:27.704655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:27.704660 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:27.704927 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:27.705680 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:27.705692 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:27.706142 (+ 450us) client_negotiation.cc:769] Sending connection context
0927 13:56:27.707861 (+ 1719us) client_negotiation.cc:240] Negotiation successful
0927 13:56:27.708626 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31870,"mutex_wait_us":38,"thread_start_us":32208,"threads_started":1}
I20260927 13:56:27.710171 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:27.655029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.193:42103)
0927 13:56:27.689717 (+ 34688us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:27.689739 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:27.689878 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:27.690483 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:27.690488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:27.691160 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:27.691468 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:27.691482 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.693059 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.693064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:27.694697 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:27.694712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.695888 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.695893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:27.696157 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:27.696773 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:27.696797 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:27.701863 (+ 5066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:27.706029 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:27.706034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:27.706037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:27.706555 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:27.706975 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:27.706980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:27.706984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:27.707194 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:27.707904 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:27.707908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:27.708040 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 13:56:27.708184 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 13:56:27.709454 (+ 1270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34546,"mutex_wait_us":930,"thread_start_us":17745,"threads_started":1}
I20260927 13:56:27.710225 29118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:27.710748 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:27.651368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:39795 (local address 127.22.26.193:55537)
0927 13:56:27.673446 (+ 22078us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:27.673515 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:27.683709 (+ 10194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:27.689416 (+ 5707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:27.689424 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:27.690074 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:27.690420 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:27.690435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.692549 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.692577 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:27.694269 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:27.694285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:27.695641 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:27.695648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:27.695875 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:27.696581 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:27.696609 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:27.700804 (+ 4195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:27.704774 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:27.704779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:27.704782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:27.705740 (+ 958us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:27.706197 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:27.706202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:27.706213 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:27.706461 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:27.707165 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:27.707169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:27.707323 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 13:56:27.708389 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0927 13:56:27.709677 (+ 1288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21634,"mutex_wait_us":972,"spinlock_wait_cycles":20736,"thread_start_us":37954,"threads_started":1}
I20260927 13:56:27.711745 29785 heartbeater.cc:347] Connected to a master server at 127.22.26.253:45111
I20260927 13:56:27.711949 29783 heartbeater.cc:347] Connected to a master server at 127.22.26.252:39795
I20260927 13:56:27.712160 29783 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:27.712168 29785 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:27.712998 29784 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39559
I20260927 13:56:27.713137 29784 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:27.713163 29785 heartbeater.cc:511] Master 127.22.26.253:45111 requested a full tablet report, sending...
I20260927 13:56:27.713609 29784 heartbeater.cc:511] Master 127.22.26.254:39559 requested a full tablet report, sending...
I20260927 13:56:27.713968 29783 heartbeater.cc:511] Master 127.22.26.252:39795 requested a full tablet report, sending...
I20260927 13:56:27.715045 28799 ts_manager.cc:194] Re-registered known tserver with Master: 88dc53435f9d4db09dc2b185d4505baf (127.22.26.193:43167)
I20260927 13:56:27.716075 28799 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:42103
I20260927 13:56:27.716179 29404 ts_manager.cc:194] Registered new tserver with Master: 88dc53435f9d4db09dc2b185d4505baf (127.22.26.193:43167)
I20260927 13:56:27.717609 29598 ts_manager.cc:194] Registered new tserver with Master: 88dc53435f9d4db09dc2b185d4505baf (127.22.26.193:43167)
I20260927 13:56:27.801378 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 28995
I20260927 13:56:27.818202 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:44945
--local_ip_for_outbound_sockets=127.22.26.194
--tserver_master_addrs=127.22.26.254:39559
--webserver_port=41073
--webserver_interface=127.22.26.194
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab
--principal=oryx/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.26.254:39559,127.22.26.253:45111
--tserver_master_addrs=127.22.26.254:39559,127.22.26.253:45111,127.22.26.252:39795 with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:28.137594 29795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:28.137909 29795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:28.138069 29795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:28.145435 29795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:28.145532 29795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:28.145648 29795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:56:28.155587 29795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab
--principal=oryx/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:44945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=41073
--tserver_master_addrs=127.22.26.254:39559,127.22.26.253:45111,127.22.26.252:39795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:28.156852 29795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:28.158089 29795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:28.171811 29800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:28.171815 29803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:28.171849 29801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:28.176642 29795 server_base.cc:1034] running on GCE node
I20260927 13:56:28.177182 29795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:28.178139 29795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:28.179365 29795 hybrid_clock.cc:648] HybridClock initialized: now 1790517388179291 us; error 62 us; skew 500 ppm
I20260927 13:56:28.179690 29795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:28 dist-test-slave-gkks krb5kdc[28755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517388, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:28.182721 29795 init.cc:377] Logged in from keytab as oryx/127.22.26.194@KRBTEST.COM (short username oryx)
I20260927 13:56:28.184120 29795 webserver.cc:460] Webserver started at http://127.22.26.194:41073/ using document root <none> and password file <none>
I20260927 13:56:28.184772 29795 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:28.184886 29795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:28.190021 29795 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 13:56:28.192677 29810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:28.193928 29795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:56:28.194058 29795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "26964d7eab3b4d0f8940d3e911c911ac"
format_stamp: "Formatted at 2026-09-27 13:56:25 on dist-test-slave-gkks"
server_key: "0e019099679f978af666c9757b2f1be3"
server_key_iv: "800b4cd07ecfb9810a1b5caf936a66fa"
server_key_version: "encryptionkey@0"
I20260927 13:56:28.194944 29795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:28.209931 29795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:28.214840 29795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:28.215147 29795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:28.216130 29795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:28.217703 29795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:28.217813 29795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:28.217912 29795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:28.217991 29795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:28.251498 29795 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:44945
I20260927 13:56:28.251581 29922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:44945 every 8 connection(s)
I20260927 13:56:28.252907 29795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:56:28.258080 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 29795
I20260927 13:56:28.258400 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 29128
Sep 27 13:56:28 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517388, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.252@KRBTEST.COM
Sep 27 13:56:28 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517388, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
Sep 27 13:56:28 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.26.195@KRBTEST.COM
Sep 27 13:56:28 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517388, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.253@KRBTEST.COM
I20260927 13:56:28.288033 29929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.256750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:41397 (local address 127.22.26.252:39795)
0927 13:56:28.257198 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:28.257206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:28.264427 (+ 7221us) server_negotiation.cc:407] Connection header received
0927 13:56:28.267309 (+ 2882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:28.267315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:28.267417 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:28.267511 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:28.270515 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.271770 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.274316 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.274949 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.275052 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.280104 (+ 5052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:28.280113 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:28.280118 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:28.280155 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:28.283990 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.284981 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.284987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.284991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.285143 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.285793 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.285798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.285802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.286169 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:28.286306 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:28.287389 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0927 13:56:28.287624 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":2115,"threads_started":1}
I20260927 13:56:28.288326 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.256390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:59161 (local address 127.22.26.254:39559)
0927 13:56:28.256810 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:28.256816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:28.267866 (+ 11050us) server_negotiation.cc:407] Connection header received
0927 13:56:28.268009 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:28.268018 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:28.268121 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:28.268256 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:28.270550 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.272076 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.274699 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.275615 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.275752 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.281350 (+ 5598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:28.281358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:28.281362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:28.281403 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:28.284881 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.285793 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.285800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.285803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.285967 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.286464 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.286471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.286477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.286871 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:28.287003 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:28.287604 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 13:56:28.287849 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":505,"threads_started":1}
I20260927 13:56:28.290609 29931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.256560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:39795 (local address 127.22.26.194:41397)
0927 13:56:28.264272 (+ 7712us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:28.264319 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:28.266923 (+ 2604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:28.268079 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:28.268090 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:28.269219 (+ 1129us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:28.270312 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.270346 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.271992 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.271997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.274150 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.274174 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.275447 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.275453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.275718 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.276585 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:28.276618 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:28.279855 (+ 3237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:28.284153 (+ 4298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:28.284175 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:28.284194 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:28.284841 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:28.285307 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:28.285312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:28.285315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:28.285655 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:28.286412 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:28.286420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:28.286891 (+ 471us) client_negotiation.cc:769] Sending connection context
0927 13:56:28.288407 (+ 1516us) client_negotiation.cc:240] Negotiation successful
0927 13:56:28.289239 (+ 832us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7080,"mutex_wait_us":380,"spinlock_wait_cycles":32128,"thread_start_us":10735,"threads_started":1}
I20260927 13:56:28.290632 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.257777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.194:59161)
0927 13:56:28.267750 (+ 9973us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:28.267768 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:28.267885 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:28.268623 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:28.268627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:28.269329 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:28.270315 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.270351 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.272316 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.272324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.274488 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.274508 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.276539 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.276552 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.276849 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.277591 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:28.277606 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:28.281093 (+ 3487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:28.285048 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:28.285053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:28.285058 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:28.285646 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:28.286134 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:28.286142 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:28.286146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:28.286332 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:28.287146 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:28.287151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:28.287307 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 13:56:28.288573 (+ 1266us) client_negotiation.cc:240] Negotiation successful
0927 13:56:28.289067 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9827,"mutex_wait_us":32,"spinlock_wait_cycles":44416,"thread_start_us":6774,"threads_started":1}
I20260927 13:56:28.292454 29924 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39559
I20260927 13:56:28.292773 29924 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:28.293745 29924 heartbeater.cc:511] Master 127.22.26.254:39559 requested a full tablet report, sending...
I20260927 13:56:28.294649 29923 heartbeater.cc:347] Connected to a master server at 127.22.26.252:39795
I20260927 13:56:28.294828 29923 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:28.294833 29934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.260044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.22.26.195:46789)
0927 13:56:28.268521 (+ 8477us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:28.268532 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:28.269645 (+ 1113us) server_negotiation.cc:407] Connection header received
0927 13:56:28.269930 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:28.269941 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:28.270187 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:28.270339 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:28.272655 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.274142 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.275996 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.276815 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.276945 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.285893 (+ 8948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:28.285911 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:28.285929 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:28.285968 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:28.290333 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.291772 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.291781 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.291786 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.292443 (+ 657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.292985 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.292990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.292993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.293428 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:28.293743 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:28.294081 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 13:56:28.294367 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8302,"thread_start_us":8126,"threads_started":1}
I20260927 13:56:28.294963 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.257144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:45111 (local address 127.22.26.194:53515)
0927 13:56:28.264884 (+ 7740us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:28.264903 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:28.266923 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:28.277775 (+ 10852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:28.277781 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:28.278277 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:28.278544 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.278560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.280044 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.280052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.281682 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.281697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.284648 (+ 2951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.284655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.284859 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.285491 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:28.285512 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:28.288173 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:28.292085 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:28.292090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:28.292093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:28.292578 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:28.293037 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:28.293042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:28.293046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:28.293207 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:28.293863 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:28.293870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:28.294039 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 13:56:28.294186 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 13:56:28.294487 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7626,"thread_start_us":6376,"threads_started":1}
I20260927 13:56:28.295349 29923 heartbeater.cc:511] Master 127.22.26.252:39795 requested a full tablet report, sending...
I20260927 13:56:28.295452 29927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.255879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:53515 (local address 127.22.26.253:45111)
0927 13:56:28.277432 (+ 21553us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:28.277438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:28.277456 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:56:28.277516 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:28.277520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:28.277596 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:28.277727 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:28.278730 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.279828 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.283756 (+ 3928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.284425 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.285159 (+ 734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.288466 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:28.288471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:28.288475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:28.288513 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:28.291960 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.292725 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.292743 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.292746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.292894 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.293335 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.293341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.293345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.293702 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:28.294748 (+ 1046us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:28.294885 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 13:56:28.295160 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21425,"thread_start_us":21330,"threads_started":1}
I20260927 13:56:28.296535 28799 ts_manager.cc:194] Re-registered known tserver with Master: 26964d7eab3b4d0f8940d3e911c911ac (127.22.26.194:44945)
I20260927 13:56:28.296697 29598 ts_manager.cc:194] Registered new tserver with Master: 26964d7eab3b4d0f8940d3e911c911ac (127.22.26.194:44945)
I20260927 13:56:28.296972 29925 heartbeater.cc:347] Connected to a master server at 127.22.26.253:45111
I20260927 13:56:28.297101 29925 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:28.297552 29925 heartbeater.cc:511] Master 127.22.26.253:45111 requested a full tablet report, sending...
I20260927 13:56:28.297740 28799 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:59161
I20260927 13:56:28.298249 29251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:28.298655 29404 ts_manager.cc:194] Registered new tserver with Master: 26964d7eab3b4d0f8940d3e911c911ac (127.22.26.194:44945)
I20260927 13:56:28.382881 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 29128
I20260927 13:56:28.400085 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:46789
--local_ip_for_outbound_sockets=127.22.26.195
--tserver_master_addrs=127.22.26.254:39559
--webserver_port=38207
--webserver_interface=127.22.26.195
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab
--principal=oryx/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.26.254:39559,127.22.26.253:45111
--tserver_master_addrs=127.22.26.254:39559,127.22.26.253:45111,127.22.26.252:39795 with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:56:28.718951 29784 heartbeater.cc:503] Master 127.22.26.254:39559 was elected leader, sending a full tablet report...
W20260927 13:56:28.722945 29936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:28.723294 29936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:28.723467 29936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:28.730638 29936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:28.730773 29936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:28.730945 29936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:56:28.740600 29936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab
--principal=oryx/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:46789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=38207
--tserver_master_addrs=127.22.26.254:39559,127.22.26.253:45111,127.22.26.252:39795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:28.741891 29936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:28.743181 29936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:28.756417 29941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:28.756465 29942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:28.756428 29944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:28.757663 29936 server_base.cc:1034] running on GCE node
I20260927 13:56:28.758248 29936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:28.759282 29936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:28.760474 29936 hybrid_clock.cc:648] HybridClock initialized: now 1790517388760421 us; error 50 us; skew 500 ppm
I20260927 13:56:28.760790 29936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:28 dist-test-slave-gkks krb5kdc[28755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517388, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:28.764180 29936 init.cc:377] Logged in from keytab as oryx/127.22.26.195@KRBTEST.COM (short username oryx)
I20260927 13:56:28.765688 29936 webserver.cc:460] Webserver started at http://127.22.26.195:38207/ using document root <none> and password file <none>
I20260927 13:56:28.766433 29936 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:28.766582 29936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:28.772172 29936 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:56:28.774710 29951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:28.776057 29936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:56:28.776240 29936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5"
format_stamp: "Formatted at 2026-09-27 13:56:25 on dist-test-slave-gkks"
server_key: "c1e4c2c57de6b4cc524bfdefc9fab690"
server_key_iv: "f1adfd27f5952ee71dc2c29697f3b91b"
server_key_version: "encryptionkey@0"
I20260927 13:56:28.777197 29936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:28.794951 29936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:28.799343 29936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:28.799618 29936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:28.800549 29936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:28.801862 29936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:28.801983 29936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:28.802117 29936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:28.802205 29936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:28.835559 29936 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:46789
I20260927 13:56:28.835655 30063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:46789 every 8 connection(s)
I20260927 13:56:28.837098 29936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:56:28.838982 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 29936
Sep 27 13:56:28 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517388, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
Sep 27 13:56:28 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517388, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.252@KRBTEST.COM
I20260927 13:56:28.900378 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.840475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:52839 (local address 127.22.26.254:39559)
0927 13:56:28.855762 (+ 15287us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:28.855769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:28.863820 (+ 8051us) server_negotiation.cc:407] Connection header received
0927 13:56:28.870817 (+ 6997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:28.870827 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:28.870959 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:28.871069 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:28.874226 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.875717 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.878811 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.879806 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.879975 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.887427 (+ 7452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:28.887433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:28.887439 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:28.887481 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:28.891593 (+ 4112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.897093 (+ 5500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.897102 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.897107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.897264 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.897929 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.897934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.897938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.898316 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:28.898450 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:28.899641 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0927 13:56:28.899912 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15149,"thread_start_us":15297,"threads_started":1}
I20260927 13:56:28.902894 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.841283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.195:52839)
0927 13:56:28.863610 (+ 22327us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:28.863672 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:28.870383 (+ 6711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:28.871840 (+ 1457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:28.871857 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:28.873006 (+ 1149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:28.873975 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.874006 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.875973 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.875981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.878606 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.878624 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.881693 (+ 3069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.881698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.881880 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.883309 (+ 1429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:28.883342 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:28.886818 (+ 3476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:28.891855 (+ 5037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:28.891867 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:28.891887 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:28.896085 (+ 4198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:28.897425 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:28.897431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:28.897436 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:28.897797 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:28.898613 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:28.898621 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:28.899171 (+ 550us) client_negotiation.cc:769] Sending connection context
0927 13:56:28.900873 (+ 1702us) client_negotiation.cc:240] Negotiation successful
0927 13:56:28.901384 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21919,"thread_start_us":27432,"threads_started":1}
I20260927 13:56:28.905251 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.840452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:42057 (local address 127.22.26.252:39795)
0927 13:56:28.866863 (+ 26411us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:28.866871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:28.866891 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:56:28.879170 (+ 12279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:28.879179 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:28.879298 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:28.879412 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:28.882317 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.883150 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.885098 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.885711 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.885831 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.896274 (+ 10443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:28.896282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:28.896286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:28.896326 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:28.900674 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.901498 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.901503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.901506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.901650 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.902107 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.902114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.902120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.902471 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:28.902604 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:28.903902 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0927 13:56:28.904129 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26293,"thread_start_us":31900,"threads_started":1}
I20260927 13:56:28.903719 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.841931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:39795 (local address 127.22.26.195:42057)
0927 13:56:28.865252 (+ 23321us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:28.865276 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:28.878963 (+ 13687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:28.879608 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:28.879616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:28.880233 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:28.880574 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.880589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.883351 (+ 2762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.883356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.884959 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.884971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.886497 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.886504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.886709 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.887386 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:28.887408 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:28.895772 (+ 8364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:28.900873 (+ 5101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:28.900878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:28.900881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:28.901340 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:28.901787 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:28.901792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:28.901795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:28.901978 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:28.902672 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:28.902677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:28.902809 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 13:56:28.902976 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 13:56:28.903304 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23175,"thread_start_us":21920,"threads_started":1}
I20260927 13:56:28.905599 30065 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39559
I20260927 13:56:28.906160 30065 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:28.906700 30064 heartbeater.cc:347] Connected to a master server at 127.22.26.252:39795
I20260927 13:56:28.906833 30064 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:28.907229 30065 heartbeater.cc:511] Master 127.22.26.254:39559 requested a full tablet report, sending...
I20260927 13:56:28.907339 30064 heartbeater.cc:511] Master 127.22.26.252:39795 requested a full tablet report, sending...
I20260927 13:56:28.907721 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.861079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51142 (local address 127.22.26.254:39559)
0927 13:56:28.875068 (+ 13989us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:28.875094 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:28.875118 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:56:28.880192 (+ 5074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:28.880200 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:28.880349 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:28.880472 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:28.885174 (+ 4702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.886280 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.888378 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.889092 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.889240 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.893135 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:28.893142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:28.893146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:28.893186 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:28.896576 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.904639 (+ 8063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.904644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.904647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.904801 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.906055 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.906060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.906062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.906402 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:28.906646 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:28.907000 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 13:56:28.907415 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13832,"thread_start_us":14032,"threads_started":1}
I20260927 13:56:28.908965 28800 ts_manager.cc:194] Re-registered known tserver with Master: 7b1eb4bf19e9413a8f3d63ff6b7583c5 (127.22.26.195:46789)
I20260927 13:56:28.909134 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.866697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35354 (local address 127.22.26.252:39795)
0927 13:56:28.883563 (+ 16866us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:28.883570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:28.883587 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:56:28.883637 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:28.883644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:28.883731 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:28.883860 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:28.886607 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.887990 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.890454 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.891245 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.891384 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.896412 (+ 5028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:28.896423 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:28.896429 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:28.896471 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:28.902446 (+ 5975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.903371 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.903379 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.903386 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.903557 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.904088 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.904093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.904097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.904445 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:28.904695 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:28.905785 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0927 13:56:28.906256 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16735,"thread_start_us":17232,"threads_started":1}
I20260927 13:56:28.909211 29598 ts_manager.cc:194] Registered new tserver with Master: 7b1eb4bf19e9413a8f3d63ff6b7583c5 (127.22.26.195:46789)
I20260927 13:56:28.910753 28800 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:52839
Sep 27 13:56:28 dist-test-slave-gkks krb5kdc[28755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517388, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.253@KRBTEST.COM
I20260927 13:56:28.911957 28800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:56:28.920778 28800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:56:28.923245 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.886399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.22.26.253:45111)
0927 13:56:28.890344 (+ 3945us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:28.890353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:28.890374 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:56:28.890431 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:28.890436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:28.890505 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:28.893146 (+ 2641us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:28.893355 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.894586 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.898236 (+ 3650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.898958 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.899740 (+ 782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.902608 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:28.902615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:28.902619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:28.902657 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:28.919793 (+ 17136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.920725 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.920734 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.920740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.920941 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.921427 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.921432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.921436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.921830 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:28.922173 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:28.922576 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 13:56:28.922815 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3819,"thread_start_us":9704,"threads_started":1}
I20260927 13:56:28.925910 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.847449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:44001 (local address 127.22.26.253:45111)
0927 13:56:28.886176 (+ 38727us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:28.886185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:28.889065 (+ 2880us) server_negotiation.cc:407] Connection header received
0927 13:56:28.889130 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:28.889135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:28.889215 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:28.889317 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:28.894769 (+ 5452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.896128 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.910670 (+ 14542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.911364 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.911485 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.915435 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:28.915445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:28.915450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:28.915503 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:28.922393 (+ 6890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.923292 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.923300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.923303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.923459 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:28.923968 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:28.923975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:28.923978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:28.924357 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:28.924498 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:28.925221 (+ 723us) server_negotiation.cc:299] Negotiation successful
0927 13:56:28.925489 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38580,"thread_start_us":38748,"threads_started":1}
I20260927 13:56:28.926199 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.846888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:45111 (local address 127.22.26.195:44001)
0927 13:56:28.888820 (+ 41932us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:28.888841 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:28.888957 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:28.889845 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:28.889849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:28.893900 (+ 4051us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:28.894605 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.894619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.897419 (+ 2800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.897425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.900661 (+ 3236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.900692 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.911785 (+ 11093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.911792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.911981 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.912615 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:28.912637 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:28.915200 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:28.922595 (+ 7395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:28.922600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:28.922603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:28.923141 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:28.923604 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:28.923611 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:28.923634 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:28.923815 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:28.924517 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:28.924522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:28.924715 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 13:56:28.924886 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 13:56:28.925806 (+ 920us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41794,"thread_start_us":42772,"threads_started":1}
I20260927 13:56:28.927526 30066 heartbeater.cc:347] Connected to a master server at 127.22.26.253:45111
I20260927 13:56:28.927685 30066 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:28.928200 30066 heartbeater.cc:511] Master 127.22.26.253:45111 requested a full tablet report, sending...
I20260927 13:56:28.935591 29404 ts_manager.cc:194] Registered new tserver with Master: 7b1eb4bf19e9413a8f3d63ff6b7583c5 (127.22.26.195:46789)
I20260927 13:56:28.955216 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.944487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:43167 (local address 127.0.0.1:52384)
0927 13:56:28.945135 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:28.945154 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:28.945280 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:28.946398 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:28.946406 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:28.946437 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:28.948153 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.948166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.950307 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.950314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.952428 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.952440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.954213 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.954222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.954426 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.954468 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 13:56:28.954575 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 13:56:28.954735 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":306,"threads_started":1}
I20260927 13:56:28.956046 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.944737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.22.26.193:43167)
0927 13:56:28.945367 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:28.945379 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:28.945412 (+ 33us) server_negotiation.cc:407] Connection header received
0927 13:56:28.945539 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:28.945551 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:28.945883 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:28.946107 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:28.948442 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.950141 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.952581 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.953979 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.954129 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.954389 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:28.955544 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0927 13:56:28.955740 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":318,"threads_started":1}
I20260927 13:56:28.957723 30100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.945987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:44945 (local address 127.0.0.1:59538)
0927 13:56:28.948748 (+ 2761us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:28.948766 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:28.948883 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:28.951189 (+ 2306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:28.951194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:28.951220 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:28.951612 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.951624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.953577 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.953583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.955519 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.955533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.957006 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.957011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.957222 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.957249 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 13:56:28.957320 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 13:56:28.957467 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2590,"thread_start_us":4283,"threads_started":1}
I20260927 13:56:28.957871 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.946328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.22.26.194:44945)
0927 13:56:28.947723 (+ 1395us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:28.947731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:28.950450 (+ 2719us) server_negotiation.cc:407] Connection header received
0927 13:56:28.950562 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:28.950569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:28.950862 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:28.951048 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:28.951856 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.953432 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.955689 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.956809 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.956961 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.957146 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:28.957416 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 13:56:28.957584 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":980,"threads_started":1}
I20260927 13:56:28.959894 29718 tablet_service.cc:1467] Processing CreateTablet for tablet 9cde59a4df7340d38e5dbc5d589c2dc1 (DEFAULT_TABLE table=test-table [id=6bce4e2e68bb4f4ebf8b39f7e18bc748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:28.961258 29718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cde59a4df7340d38e5dbc5d589c2dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:28.961300 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.945575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:46789 (local address 127.0.0.1:50730)
0927 13:56:28.948102 (+ 2527us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:28.948123 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:28.948269 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:28.948973 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:28.948979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:28.948998 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:28.949303 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.949311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.952617 (+ 3306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.952622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.954642 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:28.954657 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.960395 (+ 5738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.960402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:28.960603 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.960643 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 13:56:28.960762 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 13:56:28.960904 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2371,"mutex_wait_us":1,"thread_start_us":1989,"threads_started":1}
I20260927 13:56:28.961594 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:28.945836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.22.26.195:46789)
0927 13:56:28.947155 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:28.947168 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:28.948404 (+ 1236us) server_negotiation.cc:407] Connection header received
0927 13:56:28.948513 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:28.948523 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:28.948788 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:28.948986 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:28.949462 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.951246 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.954839 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:28.956495 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:28.956594 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:28.956736 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:28.960833 (+ 4097us) server_negotiation.cc:299] Negotiation successful
0927 13:56:28.960990 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":813,"threads_started":1}
I20260927 13:56:28.962533 29858 tablet_service.cc:1467] Processing CreateTablet for tablet 9cde59a4df7340d38e5dbc5d589c2dc1 (DEFAULT_TABLE table=test-table [id=6bce4e2e68bb4f4ebf8b39f7e18bc748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:28.964377 29858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cde59a4df7340d38e5dbc5d589c2dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:28.965232 29999 tablet_service.cc:1467] Processing CreateTablet for tablet 9cde59a4df7340d38e5dbc5d589c2dc1 (DEFAULT_TABLE table=test-table [id=6bce4e2e68bb4f4ebf8b39f7e18bc748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:28.966828 29999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cde59a4df7340d38e5dbc5d589c2dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:28.974470 30101 tablet_bootstrap.cc:492] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf: Bootstrap starting.
I20260927 13:56:28.977697 30101 tablet_bootstrap.cc:654] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:28.978674 30101 log.cc:824] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:28.981396 30103 tablet_bootstrap.cc:492] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac: Bootstrap starting.
I20260927 13:56:28.981931 30101 tablet_bootstrap.cc:492] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf: No bootstrap required, opened a new log
I20260927 13:56:28.982194 30101 ts_tablet_manager.cc:1397] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 13:56:28.983506 30104 tablet_bootstrap.cc:492] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Bootstrap starting.
I20260927 13:56:28.984916 30103 tablet_bootstrap.cc:654] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:28.986143 30103 log.cc:824] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:28.987178 30104 tablet_bootstrap.cc:654] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:28.988235 30104 log.cc:824] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:28.989220 30103 tablet_bootstrap.cc:492] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac: No bootstrap required, opened a new log
I20260927 13:56:28.989498 30103 ts_tablet_manager.cc:1397] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 13:56:28.990886 30104 tablet_bootstrap.cc:492] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: No bootstrap required, opened a new log
I20260927 13:56:28.991212 30104 ts_tablet_manager.cc:1397] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260927 13:56:28.992568 30101 raft_consensus.cc:348] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } } peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } }
I20260927 13:56:28.992996 30101 raft_consensus.cc:374] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:28.993095 30101 raft_consensus.cc:729] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88dc53435f9d4db09dc2b185d4505baf, State: Initialized, Role: FOLLOWER
I20260927 13:56:28.993718 30101 consensus_queue.cc:260] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } } peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } }
I20260927 13:56:28.995472 30101 ts_tablet_manager.cc:1428] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260927 13:56:29.000710 30103 raft_consensus.cc:348] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } } peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } }
I20260927 13:56:29.001147 30103 raft_consensus.cc:374] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:29.001250 30103 raft_consensus.cc:729] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26964d7eab3b4d0f8940d3e911c911ac, State: Initialized, Role: FOLLOWER
I20260927 13:56:29.001956 30103 consensus_queue.cc:260] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } } peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } }
I20260927 13:56:29.003739 30103 ts_tablet_manager.cc:1428] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260927 13:56:29.004076 29924 heartbeater.cc:503] Master 127.22.26.254:39559 was elected leader, sending a full tablet report...
I20260927 13:56:29.004484 30104 raft_consensus.cc:348] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } } peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } }
I20260927 13:56:29.004905 30104 raft_consensus.cc:374] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:29.005030 30104 raft_consensus.cc:729] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1eb4bf19e9413a8f3d63ff6b7583c5, State: Initialized, Role: FOLLOWER
I20260927 13:56:29.005730 30104 consensus_queue.cc:260] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } } peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } }
W20260927 13:56:29.006822 29926 tablet.cc:2369] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:56:29.008273 30104 ts_tablet_manager.cc:1428] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Time spent starting tablet: real 0.017s user 0.007s sys 0.005s
I20260927 13:56:29.008612 30065 heartbeater.cc:503] Master 127.22.26.254:39559 was elected leader, sending a full tablet report...
I20260927 13:56:29.084393 30107 raft_consensus.cc:482] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:56:29.084730 30107 raft_consensus.cc:504] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } } peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } }
I20260927 13:56:29.086380 30107 leader_election.cc:288] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b1eb4bf19e9413a8f3d63ff6b7583c5 (127.22.26.195:46789), 26964d7eab3b4d0f8940d3e911c911ac (127.22.26.194:44945)
I20260927 13:56:29.092773 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:29.086799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:41597 (local address 127.22.26.194:44945)
0927 13:56:29.087275 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:29.087282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:29.087531 (+ 249us) server_negotiation.cc:407] Connection header received
0927 13:56:29.087779 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:29.087783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:29.087844 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:29.087938 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:29.088517 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.089232 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.091032 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.091861 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.092039 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:29.092112 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:29.092383 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 13:56:29.092502 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313}
I20260927 13:56:29.092882 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:29.086907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:54887 (local address 127.22.26.195:46789)
0927 13:56:29.087381 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:29.087387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:29.087876 (+ 489us) server_negotiation.cc:407] Connection header received
0927 13:56:29.088081 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:29.088085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:29.088140 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:29.088237 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:29.088810 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.089498 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.091235 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.091969 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.092156 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:29.092231 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:29.092507 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 13:56:29.092628 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 13:56:29.092980 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:29.086796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:46789 (local address 127.22.26.193:54887)
0927 13:56:29.087760 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:29.087779 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:29.087934 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:29.088264 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:29.088268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:29.088307 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:29.088640 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:29.088648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.089648 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.089653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:29.091045 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:29.091057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.092194 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.092201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:29.092358 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:29.092380 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 13:56:29.092483 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 13:56:29.092604 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":23,"thread_start_us":671,"threads_started":1}
I20260927 13:56:29.093353 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:29.086645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:44945 (local address 127.22.26.193:41597)
0927 13:56:29.087404 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:29.087427 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:29.087623 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:29.087971 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:29.087976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:29.088000 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:29.088369 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:29.088378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.089360 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.089367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:29.090880 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:29.090891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.092064 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.092070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:29.092229 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:29.092253 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 13:56:29.092358 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 13:56:29.092479 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":44,"thread_start_us":408,"threads_started":1}
I20260927 13:56:29.093958 30019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cde59a4df7340d38e5dbc5d589c2dc1" candidate_uuid: "88dc53435f9d4db09dc2b185d4505baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" is_pre_election: true
I20260927 13:56:29.094331 29878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cde59a4df7340d38e5dbc5d589c2dc1" candidate_uuid: "88dc53435f9d4db09dc2b185d4505baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26964d7eab3b4d0f8940d3e911c911ac" is_pre_election: true
I20260927 13:56:29.094539 30019 raft_consensus.cc:2454] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88dc53435f9d4db09dc2b185d4505baf in term 0.
I20260927 13:56:29.094844 29878 raft_consensus.cc:2454] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88dc53435f9d4db09dc2b185d4505baf in term 0.
I20260927 13:56:29.095288 29665 leader_election.cc:302] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b1eb4bf19e9413a8f3d63ff6b7583c5, 88dc53435f9d4db09dc2b185d4505baf; no voters:
I20260927 13:56:29.095633 30107 raft_consensus.cc:2781] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:56:29.095803 30107 raft_consensus.cc:482] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:56:29.095917 30107 raft_consensus.cc:3037] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:29.098127 30107 raft_consensus.cc:504] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } } peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } }
I20260927 13:56:29.098775 30107 leader_election.cc:288] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [CANDIDATE]: Term 1 election: Requested vote from peers 7b1eb4bf19e9413a8f3d63ff6b7583c5 (127.22.26.195:46789), 26964d7eab3b4d0f8940d3e911c911ac (127.22.26.194:44945)
I20260927 13:56:29.099260 30019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cde59a4df7340d38e5dbc5d589c2dc1" candidate_uuid: "88dc53435f9d4db09dc2b185d4505baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5"
I20260927 13:56:29.099457 30019 raft_consensus.cc:3037] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:29.099844 29878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cde59a4df7340d38e5dbc5d589c2dc1" candidate_uuid: "88dc53435f9d4db09dc2b185d4505baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26964d7eab3b4d0f8940d3e911c911ac"
I20260927 13:56:29.100016 29878 raft_consensus.cc:3037] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:29.101806 30019 raft_consensus.cc:2454] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88dc53435f9d4db09dc2b185d4505baf in term 1.
I20260927 13:56:29.102176 29665 leader_election.cc:302] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b1eb4bf19e9413a8f3d63ff6b7583c5, 88dc53435f9d4db09dc2b185d4505baf; no voters:
I20260927 13:56:29.102201 29878 raft_consensus.cc:2454] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88dc53435f9d4db09dc2b185d4505baf in term 1.
I20260927 13:56:29.102484 30107 raft_consensus.cc:2781] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:29.102926 30107 raft_consensus.cc:686] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [term 1 LEADER]: Becoming Leader. State: Replica: 88dc53435f9d4db09dc2b185d4505baf, State: Running, Role: LEADER
I20260927 13:56:29.103468 30107 consensus_queue.cc:237] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } } peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } }
W20260927 13:56:29.106901 30067 tablet.cc:2369] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:56:29.108496 28799 catalog_manager.cc:5495] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf reported cstate change: term changed from 0 to 1, leader changed from <none> to 88dc53435f9d4db09dc2b185d4505baf (127.22.26.193). New cstate: current_term: 1 leader_uuid: "88dc53435f9d4db09dc2b185d4505baf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } health_report { overall_health: UNKNOWN } } }
I20260927 13:56:29.148309 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:29.142956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.22.26.193:43167)
0927 13:56:29.143196 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:29.143202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:29.143222 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:56:29.143423 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:29.143427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:29.143511 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:29.143662 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:56:29.144041 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.144745 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.146071 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.146527 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.146657 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:29.147188 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:56:29.147743 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:56:29.147897 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:29.147954 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 13:56:29.148029 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:56:29.156217 29878 raft_consensus.cc:1261] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac [term 1 FOLLOWER]: Refusing update from remote peer 88dc53435f9d4db09dc2b185d4505baf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:29.156243 30019 raft_consensus.cc:1261] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 1 FOLLOWER]: Refusing update from remote peer 88dc53435f9d4db09dc2b185d4505baf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:29.157155 30112 consensus_queue.cc:1035] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:29.157508 30107 consensus_queue.cc:1035] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:29.170148 30115 mvcc.cc:205] Tried to move back new op lower bound from 7333959225974476800 to 7333959225771106304. Current Snapshot: MvccSnapshot[applied={T|T < 7333959225974476800}]
I20260927 13:56:29.191429 28800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:29.191682 28800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:29.197268 28800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae445534f86344d9bf1ef1b1e0328b91: Sending DeleteTablet for 3 replicas of tablet 9cde59a4df7340d38e5dbc5d589c2dc1
I20260927 13:56:29.198486 29999 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cde59a4df7340d38e5dbc5d589c2dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:29 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:50730
I20260927 13:56:29.199033 29858 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cde59a4df7340d38e5dbc5d589c2dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:29 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:59538
I20260927 13:56:29.199908 29718 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cde59a4df7340d38e5dbc5d589c2dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:29 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:52384
I20260927 13:56:29.200968 30127 tablet_replica.cc:330] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf: stopping tablet replica
I20260927 13:56:29.201488 30127 raft_consensus.cc:2229] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:56:29.202263 30127 raft_consensus.cc:2258] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:29.204627 30127 ts_tablet_manager.cc:1905] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:29.210106 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 28768
I20260927 13:56:29.210996 28830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:29.212944 30127 ts_tablet_manager.cc:1918] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:29.213119 30127 log.cc:1196] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/9cde59a4df7340d38e5dbc5d589c2dc1
I20260927 13:56:29.213757 30127 ts_tablet_manager.cc:1939] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 88dc53435f9d4db09dc2b185d4505baf: Deleting consensus metadata
I20260927 13:56:29.214628 30125 tablet_replica.cc:330] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: stopping tablet replica
I20260927 13:56:29.215198 30125 raft_consensus.cc:2229] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:29.215662 30125 raft_consensus.cc:2258] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:29.218083 30125 ts_tablet_manager.cc:1905] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:29.224255 30125 ts_tablet_manager.cc:1918] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:29.224421 30125 log.cc:1196] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/9cde59a4df7340d38e5dbc5d589c2dc1
I20260927 13:56:29.224951 30125 ts_tablet_manager.cc:1939] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Deleting consensus metadata
I20260927 13:56:29.228310 30126 tablet_replica.cc:330] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac: stopping tablet replica
I20260927 13:56:29.228757 30126 raft_consensus.cc:2229] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:29.229264 30126 raft_consensus.cc:2258] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:29.233448 30126 ts_tablet_manager.cc:1905] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:29.234833 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:29.202694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.22.26.253:45111)
0927 13:56:29.202881 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:29.202887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:29.210373 (+ 7486us) server_negotiation.cc:407] Connection header received
0927 13:56:29.210840 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:29.210845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:29.210938 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:29.211006 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:29.216350 (+ 5344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.217421 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.219366 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.220019 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.220119 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:29.227904 (+ 7785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:29.227911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:29.227916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:29.227958 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:29.231693 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:29.232533 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:29.232538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:29.232542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:29.232674 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:29.233107 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:29.233112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:29.233116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:29.233445 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:29.233714 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:29.234291 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 13:56:29.234514 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:56:29.245280 30126 ts_tablet_manager.cc:1918] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:29.245447 30126 log.cc:1196] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/9cde59a4df7340d38e5dbc5d589c2dc1
I20260927 13:56:29.245976 30126 ts_tablet_manager.cc:1939] T 9cde59a4df7340d38e5dbc5d589c2dc1 P 26964d7eab3b4d0f8940d3e911c911ac: Deleting consensus metadata
I20260927 13:56:29.251662 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:29.209439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.22.26.252:39795)
0927 13:56:29.209634 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:29.209640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:29.236207 (+ 26567us) server_negotiation.cc:407] Connection header received
0927 13:56:29.236291 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:29.236300 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:29.236401 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:29.236519 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:29.238212 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.239245 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.241146 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.241566 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.241701 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:29.245028 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:29.245035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:29.245040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:29.245079 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:29.248715 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:29.249518 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:29.249525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:29.249531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:29.249688 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:29.250148 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:29.250153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:29.250157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:29.250514 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:29.250768 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:29.251129 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 13:56:29.251355 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:56:29.377394 28784 catalog_manager.cc:4841] TS 26964d7eab3b4d0f8940d3e911c911ac (127.22.26.194:44945): tablet 9cde59a4df7340d38e5dbc5d589c2dc1 (table test-table [id=6bce4e2e68bb4f4ebf8b39f7e18bc748]) successfully deleted
I20260927 13:56:29.377728 28785 catalog_manager.cc:4841] TS 7b1eb4bf19e9413a8f3d63ff6b7583c5 (127.22.26.195:46789): tablet 9cde59a4df7340d38e5dbc5d589c2dc1 (table test-table [id=6bce4e2e68bb4f4ebf8b39f7e18bc748]) successfully deleted
W20260927 13:56:29.377908 28785 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.26.195:46789, user_credentials={real_user=slave}} blocked reactor thread for 151095us
I20260927 13:56:29.378847 28787 catalog_manager.cc:4841] TS 88dc53435f9d4db09dc2b185d4505baf (127.22.26.193:43167): tablet 9cde59a4df7340d38e5dbc5d589c2dc1 (table test-table [id=6bce4e2e68bb4f4ebf8b39f7e18bc748]) successfully deleted
W20260927 13:56:29.379037 28787 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.26.193:43167, user_credentials={real_user=slave}} blocked reactor thread for 163581us
I20260927 13:56:29.379648 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 28768
I20260927 13:56:29.436976 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:29.413351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.22.26.253:45111)
0927 13:56:29.413738 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:29.413746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:29.418283 (+ 4537us) server_negotiation.cc:407] Connection header received
0927 13:56:29.418753 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:29.418760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:29.418842 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:29.418920 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:29.421562 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.422702 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.424699 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.425327 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.425420 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:29.430119 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:29.430125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:29.430129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:29.430172 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:29.433988 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:29.434701 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:29.434706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:29.434710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:29.434913 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:29.435325 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:29.435331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:29.435334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:29.435683 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:29.435972 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:29.436434 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 13:56:29.436636 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
I20260927 13:56:29.437525 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:29.413536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.22.26.252:39795)
0927 13:56:29.413793 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:29.413801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:29.420269 (+ 6468us) server_negotiation.cc:407] Connection header received
0927 13:56:29.420741 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:29.420749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:29.420841 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:29.420917 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:29.423540 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.424779 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.426557 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:29.427215 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:29.427337 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:29.431892 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:29.431897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:29.431900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:29.431933 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:29.434458 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:29.435258 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:29.435263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:29.435265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:29.435378 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:29.435761 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:29.435767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:29.435770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:29.436150 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:29.436415 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:29.436951 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 13:56:29.437186 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:56:30.177284 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.176531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.193:34267)
0927 13:56:30.176933 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.177042 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":213,"threads_started":1}
W20260927 13:56:30.177707 29784 heartbeater.cc:644] Failed to heartbeat to 127.22.26.254:39559 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
I20260927 13:56:30.185566 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.184712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.195:37731)
0927 13:56:30.185194 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.185377 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":267,"threads_started":1}
W20260927 13:56:30.185966 30065 heartbeater.cc:644] Failed to heartbeat to 127.22.26.254:39559 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
I20260927 13:56:30.188216 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.187504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.194:55019)
0927 13:56:30.187875 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.188005 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":199,"threads_started":1}
W20260927 13:56:30.188603 29924 heartbeater.cc:644] Failed to heartbeat to 127.22.26.254:39559 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
I20260927 13:56:30.698494 30140 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae445534f86344d9bf1ef1b1e0328b91)
I20260927 13:56:30.698776 30140 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } }
I20260927 13:56:30.699723 30140 leader_election.cc:288] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae445534f86344d9bf1ef1b1e0328b91 (127.22.26.254:39559), d9f5a277ee88440eb96a393fe2489a5b (127.22.26.253:45111)
I20260927 13:56:30.700451 30141 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae445534f86344d9bf1ef1b1e0328b91)
I20260927 13:56:30.700883 30141 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } }
I20260927 13:56:30.701326 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.699957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51606)
0927 13:56:30.700718 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.700980 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":610,"threads_started":1}
W20260927 13:56:30.702633 29584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
I20260927 13:56:30.704443 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.703883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51630)
0927 13:56:30.704069 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.704219 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 13:56:30.704672 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.703037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51618)
0927 13:56:30.704174 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.704358 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1011,"thread_start_us":491,"threads_started":1}
W20260927 13:56:30.704721 29584 leader_election.cc:334] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae445534f86344d9bf1ef1b1e0328b91 (127.22.26.254:39559): Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
W20260927 13:56:30.704949 29390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
I20260927 13:56:30.705215 30141 leader_election.cc:288] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae445534f86344d9bf1ef1b1e0328b91 (127.22.26.254:39559), bc1e0ccd48fb468e80426dec238012ff (127.22.26.252:39795)
I20260927 13:56:30.707368 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.706758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51636)
0927 13:56:30.706976 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.707197 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 13:56:30.707620 29390 leader_election.cc:334] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae445534f86344d9bf1ef1b1e0328b91 (127.22.26.254:39559): Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
I20260927 13:56:30.709908 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.699962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:45111 (local address 127.0.0.1:56902)
0927 13:56:30.701572 (+ 1610us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.701597 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:30.701779 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:30.702757 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:30.702762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:30.702812 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:30.703118 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:30.703139 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.705388 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.705396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:30.707877 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:30.707922 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.709060 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.709066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:30.709245 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:30.709269 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 13:56:30.709384 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 13:56:30.709532 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1487,"mutex_wait_us":149,"thread_start_us":322,"threads_started":1}
I20260927 13:56:30.709954 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.700111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.22.26.253:45111)
0927 13:56:30.702391 (+ 2280us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:30.702397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:30.702424 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:56:30.702497 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:30.702502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:30.702575 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:30.702691 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:30.703393 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.704905 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.708072 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.708830 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.708976 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:30.709127 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:30.709429 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 13:56:30.709597 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":2130,"thread_start_us":1882,"threads_started":1}
I20260927 13:56:30.710566 29414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc1e0ccd48fb468e80426dec238012ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d9f5a277ee88440eb96a393fe2489a5b" is_pre_election: true
I20260927 13:56:30.710845 29414 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc1e0ccd48fb468e80426dec238012ff in term 1.
I20260927 13:56:30.711328 29582 leader_election.cc:302] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc1e0ccd48fb468e80426dec238012ff, d9f5a277ee88440eb96a393fe2489a5b; no voters: ae445534f86344d9bf1ef1b1e0328b91
I20260927 13:56:30.711724 30140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 13:56:30.711843 30140 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae445534f86344d9bf1ef1b1e0328b91)
I20260927 13:56:30.711936 30140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:56:30.713382 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.704092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:39795 (local address 127.0.0.1:35374)
0927 13:56:30.705534 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.705553 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:30.705676 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:30.707603 (+ 1927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:30.707614 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:30.707644 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:30.707913 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:30.707930 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.709009 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.709014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:30.711001 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:30.711018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.712520 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.712531 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:30.712718 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:30.712751 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:56:30.712900 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 13:56:30.713052 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1356,"mutex_wait_us":21,"thread_start_us":2622,"threads_started":1}
I20260927 13:56:30.713428 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.705510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.22.26.252:39795)
0927 13:56:30.707311 (+ 1801us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:30.707317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:30.707338 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:56:30.707394 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:30.707403 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:30.707481 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:30.707581 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:30.708087 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.708852 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.711181 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.712333 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.712495 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:30.712626 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:30.712933 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 13:56:30.713107 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1677,"thread_start_us":1488,"threads_started":1}
I20260927 13:56:30.714269 29608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9f5a277ee88440eb96a393fe2489a5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bc1e0ccd48fb468e80426dec238012ff" is_pre_election: true
I20260927 13:56:30.714768 30140 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } }
I20260927 13:56:30.715466 29608 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9f5a277ee88440eb96a393fe2489a5b in current term 2: Already voted for candidate bc1e0ccd48fb468e80426dec238012ff in this term.
I20260927 13:56:30.716106 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.715666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51656)
0927 13:56:30.715820 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.715913 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 13:56:30.716367 30140 leader_election.cc:288] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [CANDIDATE]: Term 2 election: Requested vote from peers ae445534f86344d9bf1ef1b1e0328b91 (127.22.26.254:39559), d9f5a277ee88440eb96a393fe2489a5b (127.22.26.253:45111)
I20260927 13:56:30.716403 29391 leader_election.cc:302] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9f5a277ee88440eb96a393fe2489a5b; no voters: ae445534f86344d9bf1ef1b1e0328b91, bc1e0ccd48fb468e80426dec238012ff
I20260927 13:56:30.716811 30141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:56:30.717417 29414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc1e0ccd48fb468e80426dec238012ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d9f5a277ee88440eb96a393fe2489a5b"
I20260927 13:56:30.718029 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.717737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51670)
0927 13:56:30.717858 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.717935 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 13:56:30.718273 29584 leader_election.cc:334] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ae445534f86344d9bf1ef1b1e0328b91 (127.22.26.254:39559): Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
I20260927 13:56:30.719485 30141 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 13:56:30.721515 29414 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc1e0ccd48fb468e80426dec238012ff in term 2.
I20260927 13:56:30.722115 29582 leader_election.cc:302] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc1e0ccd48fb468e80426dec238012ff, d9f5a277ee88440eb96a393fe2489a5b; no voters: ae445534f86344d9bf1ef1b1e0328b91
I20260927 13:56:30.722404 30140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:56:30.722872 30140 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [term 2 LEADER]: Becoming Leader. State: Replica: bc1e0ccd48fb468e80426dec238012ff, State: Running, Role: LEADER
I20260927 13:56:30.723306 30140 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } }
I20260927 13:56:30.724840 30150 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc1e0ccd48fb468e80426dec238012ff. Latest consensus state: current_term: 2 leader_uuid: "bc1e0ccd48fb468e80426dec238012ff" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } } }
I20260927 13:56:30.725126 30150 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:30.725772 30152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:30.729286 30152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:30.730096 30152 catalog_manager.cc:1259] Loaded cluster ID: 592198b8ff06425389edd11acf820348
I20260927 13:56:30.730202 30152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:30.731415 30152 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:30.732225 30152 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: Loaded TSK: 0
I20260927 13:56:30.733062 30152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:56:30.772034 29598 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:56:30.779479 29598 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:56:30.783763 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.783371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51692)
0927 13:56:30.783557 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.783631 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 13:56:30.783981 29414 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [term 2 FOLLOWER]: Refusing update from remote peer bc1e0ccd48fb468e80426dec238012ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 13:56:30.784909 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.784510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51702)
0927 13:56:30.784646 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.784715 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 13:56:30.784906 30140 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 13:56:30.785207 29584 consensus_peers.cc:487] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff -> Peer ae445534f86344d9bf1ef1b1e0328b91 (127.22.26.254:39559): Couldn't send request to peer ae445534f86344d9bf1ef1b1e0328b91. Status: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 13:56:30.787555 30141 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc1e0ccd48fb468e80426dec238012ff. Latest consensus state: current_term: 2 leader_uuid: "bc1e0ccd48fb468e80426dec238012ff" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } } }
I20260927 13:56:30.787825 30141 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:56:30.789412 30140 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "bc1e0ccd48fb468e80426dec238012ff" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } } }
I20260927 13:56:30.789697 30140 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:30.791682 30141 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "bc1e0ccd48fb468e80426dec238012ff" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae445534f86344d9bf1ef1b1e0328b91" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39559 } } peers { permanent_uuid: "d9f5a277ee88440eb96a393fe2489a5b" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 45111 } attrs { promote: false } } peers { permanent_uuid: "bc1e0ccd48fb468e80426dec238012ff" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 39795 } attrs { promote: false } } }
I20260927 13:56:30.791978 30141 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f5a277ee88440eb96a393fe2489a5b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:56:30.804360 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.797730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.22.26.194:44945)
0927 13:56:30.798170 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:30.798176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:30.798200 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:56:30.798279 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:30.798284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:30.798360 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:30.798495 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:30.798864 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.800020 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.801902 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.803156 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.803288 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:30.803391 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:30.803886 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 13:56:30.804033 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":236,"threads_started":1}
I20260927 13:56:30.804484 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.797389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:44945 (local address 127.0.0.1:59542)
0927 13:56:30.797548 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.797566 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:30.797672 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:30.798507 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:30.798512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:30.798536 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:30.798724 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:30.798733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.800176 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.800183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:30.801682 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:30.801708 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.803338 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.803347 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:30.803654 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:30.803716 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 13:56:30.803848 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 13:56:30.804046 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 13:56:30.805315 29858 tablet_service.cc:1467] Processing CreateTablet for tablet 789630c4cf9e42edacf8c60a4386a253 (DEFAULT_TABLE table=test-table [id=6e058f233e88470bbaf5f2fe0b696eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:30.805807 29858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789630c4cf9e42edacf8c60a4386a253. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:30.806780 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.798012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.22.26.193:43167)
0927 13:56:30.799200 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:30.799208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:30.799227 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:56:30.799359 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:30.799364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:30.799453 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:30.799566 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:30.800077 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.801326 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.803738 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.805253 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.805735 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:30.805941 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:30.806327 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 13:56:30.806489 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":1553,"threads_started":1}
I20260927 13:56:30.806933 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.798056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:43167 (local address 127.0.0.1:52392)
0927 13:56:30.798200 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.798218 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:30.798330 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:30.799597 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:30.799602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:30.799627 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:30.799901 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:30.799911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.801494 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.801500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:30.803563 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:30.803575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.805502 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.805508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:30.806117 (+ 609us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:30.806170 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 13:56:30.806268 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 13:56:30.806469 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":334}
I20260927 13:56:30.807873 29718 tablet_service.cc:1467] Processing CreateTablet for tablet 789630c4cf9e42edacf8c60a4386a253 (DEFAULT_TABLE table=test-table [id=6e058f233e88470bbaf5f2fe0b696eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:30.808377 29718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789630c4cf9e42edacf8c60a4386a253. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:30.813011 30163 tablet_bootstrap.cc:492] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac: Bootstrap starting.
I20260927 13:56:30.813196 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.799008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:46789 (local address 127.0.0.1:50746)
0927 13:56:30.804782 (+ 5774us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.804806 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:30.804960 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:30.805487 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:30.805494 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:30.805526 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:30.805813 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:30.805827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.809249 (+ 3422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.809258 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:30.811169 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:30.811185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.812275 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.812280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:30.812493 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:30.812517 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 13:56:30.812648 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 13:56:30.812807 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5637,"mutex_wait_us":1,"thread_start_us":10629,"threads_started":1}
I20260927 13:56:30.814286 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.799425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.22.26.195:46789)
0927 13:56:30.800602 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:30.800608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:30.805097 (+ 4489us) server_negotiation.cc:407] Connection header received
0927 13:56:30.805196 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:30.805203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:30.805318 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:30.807320 (+ 2002us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:30.807518 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.809050 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.811295 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:30.812095 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:30.812264 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:30.812377 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:30.813773 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0927 13:56:30.813949 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1061,"thread_start_us":966,"threads_started":1}
I20260927 13:56:30.814941 30163 tablet_bootstrap.cc:654] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:30.815034 30164 tablet_bootstrap.cc:492] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf: Bootstrap starting.
I20260927 13:56:30.815186 29999 tablet_service.cc:1467] Processing CreateTablet for tablet 789630c4cf9e42edacf8c60a4386a253 (DEFAULT_TABLE table=test-table [id=6e058f233e88470bbaf5f2fe0b696eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:30.815627 29999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789630c4cf9e42edacf8c60a4386a253. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:30.816663 30164 tablet_bootstrap.cc:654] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:30.817243 30163 tablet_bootstrap.cc:492] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac: No bootstrap required, opened a new log
I20260927 13:56:30.817373 30163 ts_tablet_manager.cc:1397] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 13:56:30.818142 30163 raft_consensus.cc:348] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } } peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } }
I20260927 13:56:30.818367 30163 raft_consensus.cc:374] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:30.818444 30163 raft_consensus.cc:729] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26964d7eab3b4d0f8940d3e911c911ac, State: Initialized, Role: FOLLOWER
I20260927 13:56:30.818666 30163 consensus_queue.cc:260] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } } peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } }
I20260927 13:56:30.819523 30163 ts_tablet_manager.cc:1428] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 13:56:30.821048 30168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.820106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.194:59241)
0927 13:56:30.820648 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.820822 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"threads_started":1}
I20260927 13:56:30.822763 30169 tablet_bootstrap.cc:492] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Bootstrap starting.
I20260927 13:56:30.822863 29923 heartbeater.cc:503] Master 127.22.26.252:39795 was elected leader, sending a full tablet report...
I20260927 13:56:30.823626 30164 tablet_bootstrap.cc:492] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf: No bootstrap required, opened a new log
I20260927 13:56:30.823871 30164 ts_tablet_manager.cc:1397] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260927 13:56:30.824342 30168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.823888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.194:59787)
0927 13:56:30.824099 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.824198 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 13:56:30.824752 30164 raft_consensus.cc:348] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } } peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } }
I20260927 13:56:30.824992 30164 raft_consensus.cc:374] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:30.825091 30164 raft_consensus.cc:729] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88dc53435f9d4db09dc2b185d4505baf, State: Initialized, Role: FOLLOWER
I20260927 13:56:30.825309 30164 consensus_queue.cc:260] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } } peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } }
I20260927 13:56:30.825551 30169 tablet_bootstrap.cc:654] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:30.826051 30164 ts_tablet_manager.cc:1428] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 13:56:30.827955 29783 heartbeater.cc:503] Master 127.22.26.252:39795 was elected leader, sending a full tablet report...
I20260927 13:56:30.829041 30169 tablet_bootstrap.cc:492] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: No bootstrap required, opened a new log
I20260927 13:56:30.829201 30169 ts_tablet_manager.cc:1397] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260927 13:56:30.830065 30169 raft_consensus.cc:348] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } } peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } }
I20260927 13:56:30.830294 30169 raft_consensus.cc:374] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:30.830405 30169 raft_consensus.cc:729] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1eb4bf19e9413a8f3d63ff6b7583c5, State: Initialized, Role: FOLLOWER
I20260927 13:56:30.830651 30169 consensus_queue.cc:260] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } } peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } }
I20260927 13:56:30.831429 30169 ts_tablet_manager.cc:1428] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 13:56:30.833419 30064 heartbeater.cc:503] Master 127.22.26.252:39795 was elected leader, sending a full tablet report...
I20260927 13:56:30.834197 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.831853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.195:45719)
0927 13:56:30.833771 (+ 1918us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.833942 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1759,"thread_start_us":1572,"threads_started":1}
I20260927 13:56:30.835335 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.834899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.195:45999)
0927 13:56:30.835123 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.835210 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 13:56:30.835458 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.826367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.193:55563)
0927 13:56:30.835069 (+ 8702us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.835226 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8552,"thread_start_us":8623,"threads_started":1}
I20260927 13:56:30.836367 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:30.836003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.193:50997)
0927 13:56:30.836157 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:30.836238 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 13:56:30.944775 29786 tablet.cc:2369] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:56:31.091856 30173 raft_consensus.cc:482] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:56:31.092131 30173 raft_consensus.cc:504] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } } peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } }
I20260927 13:56:31.093705 30173 leader_election.cc:288] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26964d7eab3b4d0f8940d3e911c911ac (127.22.26.194:44945), 88dc53435f9d4db09dc2b185d4505baf (127.22.26.193:43167)
I20260927 13:56:31.099253 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.094161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:55643 (local address 127.22.26.193:43167)
0927 13:56:31.094370 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:31.094375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:31.094400 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:56:31.094500 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:31.094504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:31.094570 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:31.094683 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:31.095170 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.095847 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.097484 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.098331 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.098513 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:31.098589 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:31.098845 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 13:56:31.098996 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 13:56:31.099339 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.093977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:43167 (local address 127.22.26.195:55643)
0927 13:56:31.094161 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:31.094182 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:31.094346 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:31.094707 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:31.094712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:31.094753 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:31.094993 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:31.095001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.095987 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.095995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:31.097329 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:31.097340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.098514 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.098520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:31.098685 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:31.098708 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 13:56:31.098820 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 13:56:31.098958 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 13:56:31.100162 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.094139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:34681 (local address 127.22.26.194:44945)
0927 13:56:31.094378 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:31.094386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:31.094934 (+ 548us) server_negotiation.cc:407] Connection header received
0927 13:56:31.095192 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:31.095197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:31.095256 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:31.095362 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:31.095779 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.096448 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.098076 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.099048 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.099235 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:31.099322 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:31.099805 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 13:56:31.099930 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 13:56:31.100265 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.093977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:44945 (local address 127.22.26.195:34681)
0927 13:56:31.094825 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:31.094839 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:31.095007 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:31.095374 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:31.095378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:31.095399 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:31.095638 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:31.095645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.096593 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.096598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:31.097926 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:31.097937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.099446 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.099457 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:31.099652 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:31.099673 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 13:56:31.099780 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 13:56:31.099897 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":258,"thread_start_us":314,"threads_started":1}
I20260927 13:56:31.100242 29738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789630c4cf9e42edacf8c60a4386a253" candidate_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88dc53435f9d4db09dc2b185d4505baf" is_pre_election: true
I20260927 13:56:31.100658 29738 raft_consensus.cc:2454] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b1eb4bf19e9413a8f3d63ff6b7583c5 in term 0.
I20260927 13:56:31.100909 29878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789630c4cf9e42edacf8c60a4386a253" candidate_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26964d7eab3b4d0f8940d3e911c911ac" is_pre_election: true
I20260927 13:56:31.101148 29878 raft_consensus.cc:2454] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b1eb4bf19e9413a8f3d63ff6b7583c5 in term 0.
I20260927 13:56:31.101227 29955 leader_election.cc:302] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b1eb4bf19e9413a8f3d63ff6b7583c5, 88dc53435f9d4db09dc2b185d4505baf; no voters:
I20260927 13:56:31.101678 30173 raft_consensus.cc:2781] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:56:31.101814 30173 raft_consensus.cc:482] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:56:31.101933 30173 raft_consensus.cc:3037] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:31.104336 30173 raft_consensus.cc:504] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } } peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } }
I20260927 13:56:31.104935 30173 leader_election.cc:288] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [CANDIDATE]: Term 1 election: Requested vote from peers 26964d7eab3b4d0f8940d3e911c911ac (127.22.26.194:44945), 88dc53435f9d4db09dc2b185d4505baf (127.22.26.193:43167)
I20260927 13:56:31.105285 29738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789630c4cf9e42edacf8c60a4386a253" candidate_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88dc53435f9d4db09dc2b185d4505baf"
I20260927 13:56:31.105288 29878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789630c4cf9e42edacf8c60a4386a253" candidate_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26964d7eab3b4d0f8940d3e911c911ac"
I20260927 13:56:31.105471 29878 raft_consensus.cc:3037] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:31.105463 29738 raft_consensus.cc:3037] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:31.107527 29878 raft_consensus.cc:2454] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b1eb4bf19e9413a8f3d63ff6b7583c5 in term 1.
I20260927 13:56:31.107591 29738 raft_consensus.cc:2454] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b1eb4bf19e9413a8f3d63ff6b7583c5 in term 1.
I20260927 13:56:31.107951 29952 leader_election.cc:302] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26964d7eab3b4d0f8940d3e911c911ac, 7b1eb4bf19e9413a8f3d63ff6b7583c5; no voters:
I20260927 13:56:31.108248 30173 raft_consensus.cc:2781] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:31.108637 30173 raft_consensus.cc:686] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 1 LEADER]: Becoming Leader. State: Replica: 7b1eb4bf19e9413a8f3d63ff6b7583c5, State: Running, Role: LEADER
I20260927 13:56:31.109033 30173 consensus_queue.cc:237] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } } peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } }
I20260927 13:56:31.109957 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.109512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.195:55651)
0927 13:56:31.109679 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:31.109810 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 13:56:31.112653 29597 catalog_manager.cc:5495] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b1eb4bf19e9413a8f3d63ff6b7583c5 (127.22.26.195). New cstate: current_term: 1 leader_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b1eb4bf19e9413a8f3d63ff6b7583c5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 46789 } health_report { overall_health: HEALTHY } } }
I20260927 13:56:31.185405 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.179929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.22.26.195:46789)
0927 13:56:31.180160 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:31.180167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:31.180187 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:56:31.180326 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:31.180331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:31.180458 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:31.180612 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:56:31.181011 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.181849 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.183209 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.183644 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.183803 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:31.184213 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:56:31.184762 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:56:31.184927 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:31.185042 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 13:56:31.185124 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 13:56:31.189666 29878 raft_consensus.cc:1261] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac [term 1 FOLLOWER]: Refusing update from remote peer 7b1eb4bf19e9413a8f3d63ff6b7583c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:31.189996 29738 raft_consensus.cc:1261] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf [term 1 FOLLOWER]: Refusing update from remote peer 7b1eb4bf19e9413a8f3d63ff6b7583c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:31.190363 30176 consensus_queue.cc:1035] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26964d7eab3b4d0f8940d3e911c911ac" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:31.190802 30173 consensus_queue.cc:1035] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88dc53435f9d4db09dc2b185d4505baf" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 43167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:31.193077 30168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.192559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.194:43039)
0927 13:56:31.192770 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:31.192866 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 13:56:31.194096 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.193062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.193:45933)
0927 13:56:31.193712 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:31.193819 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531}
I20260927 13:56:31.197556 30179 mvcc.cc:205] Tried to move back new op lower bound from 7333959234305388544 to 7333959233987084288. Current Snapshot: MvccSnapshot[applied={T|T < 7333959234305388544}]
I20260927 13:56:31.198782 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.197776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.195:60077)
0927 13:56:31.198479 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:31.198608 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528}
I20260927 13:56:31.200989 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.198514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.193:46795)
0927 13:56:31.200764 (+ 2250us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:31.200851 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2146}
I20260927 13:56:31.214464 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.213954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.195:49227)
0927 13:56:31.214164 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:31.214269 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 13:56:31.216404 30168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.215965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.22.26.194:35869)
0927 13:56:31.216175 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:31.216259 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 13:56:31.234179 29598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:31.234478 29598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:31.240321 29598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc1e0ccd48fb468e80426dec238012ff: Sending DeleteTablet for 3 replicas of tablet 789630c4cf9e42edacf8c60a4386a253
I20260927 13:56:31.241465 29718 tablet_service.cc:1514] Processing DeleteTablet for tablet 789630c4cf9e42edacf8c60a4386a253 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:31 UTC) from {username='oryx', principal='oryx/127.22.26.252@KRBTEST.COM'} at 127.0.0.1:52392
I20260927 13:56:31.241801 29999 tablet_service.cc:1514] Processing DeleteTablet for tablet 789630c4cf9e42edacf8c60a4386a253 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:31 UTC) from {username='oryx', principal='oryx/127.22.26.252@KRBTEST.COM'} at 127.0.0.1:50746
I20260927 13:56:31.242163 30189 tablet_replica.cc:330] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf: stopping tablet replica
I20260927 13:56:31.242370 30189 raft_consensus.cc:2229] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:31.242580 29858 tablet_service.cc:1514] Processing DeleteTablet for tablet 789630c4cf9e42edacf8c60a4386a253 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:31 UTC) from {username='oryx', principal='oryx/127.22.26.252@KRBTEST.COM'} at 127.0.0.1:59542
I20260927 13:56:31.242677 30189 raft_consensus.cc:2258] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:31.243299 30191 tablet_replica.cc:330] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac: stopping tablet replica
I20260927 13:56:31.243536 30191 raft_consensus.cc:2229] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:31.243886 30191 raft_consensus.cc:2258] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:31.244107 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 29570
I20260927 13:56:31.244333 30189 ts_tablet_manager.cc:1905] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:31.244894 30191 ts_tablet_manager.cc:1905] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:31.251296 30191 ts_tablet_manager.cc:1918] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:31.251458 30189 ts_tablet_manager.cc:1918] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:31.251511 30191 log.cc:1196] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/789630c4cf9e42edacf8c60a4386a253
I20260927 13:56:31.251622 30189 log.cc:1196] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/789630c4cf9e42edacf8c60a4386a253
I20260927 13:56:31.252056 30191 ts_tablet_manager.cc:1939] T 789630c4cf9e42edacf8c60a4386a253 P 26964d7eab3b4d0f8940d3e911c911ac: Deleting consensus metadata
I20260927 13:56:31.252202 30189 ts_tablet_manager.cc:1939] T 789630c4cf9e42edacf8c60a4386a253 P 88dc53435f9d4db09dc2b185d4505baf: Deleting consensus metadata
I20260927 13:56:31.253209 29582 catalog_manager.cc:4841] TS 26964d7eab3b4d0f8940d3e911c911ac (127.22.26.194:44945): tablet 789630c4cf9e42edacf8c60a4386a253 (table test-table [id=6e058f233e88470bbaf5f2fe0b696eb1]) successfully deleted
I20260927 13:56:31.253269 29585 catalog_manager.cc:4841] TS 88dc53435f9d4db09dc2b185d4505baf (127.22.26.193:43167): tablet 789630c4cf9e42edacf8c60a4386a253 (table test-table [id=6e058f233e88470bbaf5f2fe0b696eb1]) successfully deleted
I20260927 13:56:31.256781 30190 tablet_replica.cc:330] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: stopping tablet replica
I20260927 13:56:31.256980 30190 raft_consensus.cc:2229] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:56:31.257710 30190 raft_consensus.cc:2258] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:31.258920 30190 ts_tablet_manager.cc:1905] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:31.266588 30190 ts_tablet_manager.cc:1918] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:31.266868 30190 log.cc:1196] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/789630c4cf9e42edacf8c60a4386a253
I20260927 13:56:31.267534 30190 ts_tablet_manager.cc:1939] T 789630c4cf9e42edacf8c60a4386a253 P 7b1eb4bf19e9413a8f3d63ff6b7583c5: Deleting consensus metadata
I20260927 13:56:31.268769 29583 catalog_manager.cc:4841] TS 7b1eb4bf19e9413a8f3d63ff6b7583c5 (127.22.26.195:46789): tablet 789630c4cf9e42edacf8c60a4386a253 (table test-table [id=6e058f233e88470bbaf5f2fe0b696eb1]) successfully deleted
I20260927 13:56:31.278827 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.245126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.22.26.193:43167)
0927 13:56:31.245315 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:31.245322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:31.262074 (+ 16752us) server_negotiation.cc:407] Connection header received
0927 13:56:31.262302 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:31.262307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:31.262411 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:31.262515 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:31.264480 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.265614 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.267515 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.268115 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.268227 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:31.271935 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:31.271952 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:31.271971 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:31.272014 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:31.276021 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:31.276746 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:31.276755 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:31.276759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:31.276911 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:31.277275 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:31.277280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:31.277284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:31.277718 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:31.278018 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:31.278227 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 13:56:31.278447 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:56:31.279644 29778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:31.285300 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.284780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51708)
0927 13:56:31.284982 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:31.285085 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 13:56:31.286706 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.286304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:51718)
0927 13:56:31.286473 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:31.286567 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 13:56:31.404819 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 29570
I20260927 13:56:31.429384 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 29795
I20260927 13:56:31.444922 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.430204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.22.26.194:44945)
0927 13:56:31.430392 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:31.430399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:31.430420 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:56:31.430551 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:31.430557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:31.430640 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:31.430748 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:31.432098 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.433231 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.434402 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.435044 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.435319 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:31.438018 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:31.438037 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:31.438058 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:31.438098 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:31.441889 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:31.442606 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:31.442616 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:31.442619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:31.442780 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:31.443339 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:31.443344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:31.443348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:31.443803 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:31.444115 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:31.444330 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 13:56:31.444560 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:56:31.445726 29918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:31.557763 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 29795
I20260927 13:56:31.581629 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 29936
I20260927 13:56:31.597546 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.582522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.22.26.195:46789)
0927 13:56:31.582766 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:31.582772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:31.582794 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:56:31.582885 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:31.582891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:31.582970 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:31.583137 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:31.584473 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.585664 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.586834 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:31.587510 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:31.587640 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:31.590236 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:31.590252 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:31.590274 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:31.590326 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:31.594495 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:31.595207 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:31.595216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:31.595219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:31.595392 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:31.595756 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:31.595762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:31.595765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:31.596255 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:31.596591 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:31.596755 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 13:56:31.597161 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 13:56:31.598354 30059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:31.682721 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.682278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:35506)
0927 13:56:31.682464 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:31.682552 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 13:56:31.683775 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:31.683424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39559 (local address 127.0.0.1:35514)
0927 13:56:31.683568 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:31.683650 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 13:56:31.728696 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 29936
I20260927 13:56:31.751271 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 29261
I20260927 13:56:31.752436 29434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:31.871579 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 29261
I20260927 13:56:31.892809 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 29362
I20260927 13:56:31.893996 29628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:32.017678 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 29362
2026-09-27T13:56:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9716 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:56:32 dist-test-slave-gkks krb5kdc[30201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:56:32 dist-test-slave-gkks krb5kdc[30201](info): set up 2 sockets
Sep 27 13:56:32 dist-test-slave-gkks krb5kdc[30201](info): commencing operation
krb5kdc: starting...
W20260927 13:56:33.120239 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:56:33 dist-test-slave-gkks krb5kdc[30201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:56:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.254.keytab.
Entry for principal oryx/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.254.keytab.
I20260927 13:56:33.335347 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:43645
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.254.keytab
--principal=oryx/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:43645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:33.664166 30214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:33.664474 30214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:33.664633 30214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:33.672382 30214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:33.672562 30214 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 13:56:33.672650 30214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:33.672766 30214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:33.672856 30214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:33.672962 30214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:56:33.683058 30214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:43645
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.254.keytab
--principal=oryx/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:33.684487 30214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:33.685873 30214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:33.698726 30222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:33.698825 30219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:33.698797 30220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:33.700094 30214 server_base.cc:1034] running on GCE node
I20260927 13:56:33.700941 30214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:33.702327 30214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:33.703555 30214 hybrid_clock.cc:648] HybridClock initialized: now 1790517393703494 us; error 64 us; skew 500 ppm
I20260927 13:56:33.703925 30214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:33 dist-test-slave-gkks krb5kdc[30201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517393, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:33.706951 30214 init.cc:377] Logged in from keytab as oryx/127.22.26.254@KRBTEST.COM (short username oryx)
I20260927 13:56:33.708403 30214 webserver.cc:460] Webserver started at http://127.22.26.254:37671/ using document root <none> and password file <none>
I20260927 13:56:33.709110 30214 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:33.709240 30214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:33.709560 30214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:33.712503 30214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "63065962bd344208bfbd4a38a19a6100"
format_stamp: "Formatted at 2026-09-27 13:56:33 on dist-test-slave-gkks"
server_key: "377eac85867863d5db1e4fbe9d16f8e5"
server_key_iv: "23657f3b5843597e6b07e142f9d8d3af"
server_key_version: "encryptionkey@0"
I20260927 13:56:33.713295 30214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "63065962bd344208bfbd4a38a19a6100"
format_stamp: "Formatted at 2026-09-27 13:56:33 on dist-test-slave-gkks"
server_key: "377eac85867863d5db1e4fbe9d16f8e5"
server_key_iv: "23657f3b5843597e6b07e142f9d8d3af"
server_key_version: "encryptionkey@0"
I20260927 13:56:33.718201 30214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:56:33.721590 30229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:33.722898 30214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:56:33.723114 30214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "63065962bd344208bfbd4a38a19a6100"
format_stamp: "Formatted at 2026-09-27 13:56:33 on dist-test-slave-gkks"
server_key: "377eac85867863d5db1e4fbe9d16f8e5"
server_key_iv: "23657f3b5843597e6b07e142f9d8d3af"
server_key_version: "encryptionkey@0"
I20260927 13:56:33.723349 30214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:33.738267 30214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:33.742053 30214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:33.742327 30214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:33.763892 30214 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:43645
I20260927 13:56:33.763927 30290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:43645 every 8 connection(s)
I20260927 13:56:33.765321 30214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:56:33.769096 30291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:33.775000 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 30214
I20260927 13:56:33.775273 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:56:33.775921 22635 external_mini_cluster.cc:1428] Setting key 1d5486afac5249fff1346594b73cd2cf
I20260927 13:56:33.779933 30291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100: Bootstrap starting.
I20260927 13:56:33.783735 30291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:33.784796 30291 log.cc:824] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:33.788130 30291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100: No bootstrap required, opened a new log
Sep 27 13:56:33 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
I20260927 13:56:33.796478 30291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63065962bd344208bfbd4a38a19a6100" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43645 } }
I20260927 13:56:33.796919 30291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:33.797048 30291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63065962bd344208bfbd4a38a19a6100, State: Initialized, Role: FOLLOWER
I20260927 13:56:33.797712 30291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63065962bd344208bfbd4a38a19a6100" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43645 } }
I20260927 13:56:33.797971 30291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:56:33.798117 30291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:56:33.798316 30291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:33.800302 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:33.777948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.22.26.254:43645)
0927 13:56:33.778764 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:33.778776 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:33.778816 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:56:33.780539 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:33.780561 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:33.780936 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:33.781242 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:33.783189 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:33.784750 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:33.786354 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:33.787185 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:33.787348 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:33.792160 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:33.792168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:33.792184 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:33.792226 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:33.795767 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:33.796576 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:33.796586 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:33.796592 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:33.796777 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:33.797256 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:33.797263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:33.797267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:33.797840 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:33.798218 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:33.798648 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 13:56:33.799207 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":391,"threads_started":1}
I20260927 13:56:33.800910 30291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63065962bd344208bfbd4a38a19a6100" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43645 } }
I20260927 13:56:33.801479 30291 leader_election.cc:302] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63065962bd344208bfbd4a38a19a6100; no voters:
I20260927 13:56:33.802147 30291 leader_election.cc:288] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:56:33.802354 30296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:33.806610 30296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [term 1 LEADER]: Becoming Leader. State: Replica: 63065962bd344208bfbd4a38a19a6100, State: Running, Role: LEADER
I20260927 13:56:33.807674 30296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63065962bd344208bfbd4a38a19a6100" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43645 } }
I20260927 13:56:33.807919 30291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:33.810487 30298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63065962bd344208bfbd4a38a19a6100. Latest consensus state: current_term: 1 leader_uuid: "63065962bd344208bfbd4a38a19a6100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63065962bd344208bfbd4a38a19a6100" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43645 } } }
I20260927 13:56:33.810882 30298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:33.811430 30297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63065962bd344208bfbd4a38a19a6100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63065962bd344208bfbd4a38a19a6100" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43645 } } }
I20260927 13:56:33.811744 30297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100 [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:33.816383 30302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:33.821074 30302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:33.830865 30302 catalog_manager.cc:1347] Generated new cluster ID: 75da8c51714f4e5f8df07d1ceb9e5a83
I20260927 13:56:33.831001 30302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:33.845911 30302 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:56:33.847995 30302 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:33.859786 30302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100: Generated new TSK 0
I20260927 13:56:33.860891 30302 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab.
Entry for principal oryx/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab.
I20260927 13:56:33.923148 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:43645
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab
--principal=oryx/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:34.255478 30317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:34.255810 30317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:34.255990 30317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:34.263309 30317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:34.263461 30317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:34.263629 30317 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:56:34.263793 30317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:56:34.273336 30317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab
--principal=oryx/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.26.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:34.274617 30317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:34.275964 30317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:34.288717 30322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:34.288801 30323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:34.288836 30325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:34.290104 30317 server_base.cc:1034] running on GCE node
I20260927 13:56:34.290730 30317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:34.291765 30317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:34.293046 30317 hybrid_clock.cc:648] HybridClock initialized: now 1790517394292979 us; error 36 us; skew 500 ppm
I20260927 13:56:34.293365 30317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:34 dist-test-slave-gkks krb5kdc[30201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517394, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:34.303762 30317 init.cc:377] Logged in from keytab as oryx/127.22.26.193@KRBTEST.COM (short username oryx)
I20260927 13:56:34.305330 30317 webserver.cc:460] Webserver started at http://127.22.26.193:32781/ using document root <none> and password file <none>
I20260927 13:56:34.306077 30317 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:34.306166 30317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:34.306435 30317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:34.309636 30317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "ebffa94065a949fdbad26729946ae3f5"
format_stamp: "Formatted at 2026-09-27 13:56:34 on dist-test-slave-gkks"
server_key: "339307d61b141f9eede41394cb4cbd39"
server_key_iv: "7b0a78192a351a7c916892a7c66e2612"
server_key_version: "encryptionkey@0"
I20260927 13:56:34.310391 30317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "ebffa94065a949fdbad26729946ae3f5"
format_stamp: "Formatted at 2026-09-27 13:56:34 on dist-test-slave-gkks"
server_key: "339307d61b141f9eede41394cb4cbd39"
server_key_iv: "7b0a78192a351a7c916892a7c66e2612"
server_key_version: "encryptionkey@0"
I20260927 13:56:34.315862 30317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 13:56:34.319351 30332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:34.320744 30317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:56:34.320935 30317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "ebffa94065a949fdbad26729946ae3f5"
format_stamp: "Formatted at 2026-09-27 13:56:34 on dist-test-slave-gkks"
server_key: "339307d61b141f9eede41394cb4cbd39"
server_key_iv: "7b0a78192a351a7c916892a7c66e2612"
server_key_version: "encryptionkey@0"
I20260927 13:56:34.321163 30317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:34.345681 30317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:34.350140 30317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:34.350474 30317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:34.353507 30317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:34.353646 30317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:34.353816 30317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:34.353901 30317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:56:34 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517394, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
I20260927 13:56:34.377580 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:34.353881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:44253 (local address 127.22.26.254:43645)
0927 13:56:34.354394 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:34.354400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:34.355602 (+ 1202us) server_negotiation.cc:407] Connection header received
0927 13:56:34.357619 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:34.357629 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:34.357757 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:34.357896 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:34.360041 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.361077 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.363836 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.364520 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.364674 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:34.370352 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:34.370357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:34.370360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:34.370393 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:34.373692 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:34.374709 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:34.374714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:34.374717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:34.374844 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:34.375379 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:34.375384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:34.375387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:34.375796 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:34.375964 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:34.376987 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0927 13:56:34.377217 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":275,"threads_started":1}
I20260927 13:56:34.378557 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:34.354523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43645 (local address 127.22.26.193:44253)
0927 13:56:34.355407 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:34.355462 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:34.357317 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:34.358115 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:34.358123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:34.358777 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:34.359760 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:34.359781 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.361232 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.361238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:34.363695 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:34.363712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.364807 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.364817 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:34.365044 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:34.366082 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:34.366137 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:34.370076 (+ 3939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:34.373851 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:34.373860 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:34.373879 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:34.374557 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:34.374962 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:34.374967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:34.374971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:34.375257 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:34.375933 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:34.375944 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:34.376499 (+ 555us) client_negotiation.cc:769] Sending connection context
0927 13:56:34.376874 (+ 375us) client_negotiation.cc:240] Negotiation successful
0927 13:56:34.377382 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":364,"threads_started":1}
I20260927 13:56:34.403271 30412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:34.386462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43645 (local address 127.22.26.193:35523)
0927 13:56:34.387171 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:34.387193 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:34.387358 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:34.387800 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:34.387808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:34.388643 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:34.389190 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:34.389210 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.390269 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.390279 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:34.392659 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:34.392688 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.393616 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.393629 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:34.393918 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:34.394908 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:34.394942 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:34.396341 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:34.399892 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:34.399898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:34.399902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:34.400573 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:34.401026 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:34.401032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:34.401036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:34.401196 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:34.401963 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:34.401973 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:34.402247 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 13:56:34.402398 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 13:56:34.402734 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":395,"threads_started":1}
I20260927 13:56:34.404069 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:34.386436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:35523 (local address 127.22.26.254:43645)
0927 13:56:34.386680 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:34.386690 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:34.387286 (+ 596us) server_negotiation.cc:407] Connection header received
0927 13:56:34.387515 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:34.387521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:34.387620 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:34.387723 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:34.389389 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.390097 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.392854 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.393285 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.393495 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:34.396550 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:34.396554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:34.396558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:34.396598 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:34.399764 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:34.400730 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:34.400735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:34.400738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:34.400896 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:34.401380 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:34.401387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:34.401392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:34.401800 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:34.401952 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:34.402715 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 13:56:34.403718 (+ 1003us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 13:56:34.412052 30317 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:46017
I20260927 13:56:34.412081 30452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:46017 every 8 connection(s)
I20260927 13:56:34.413328 30317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:56:34.415693 30453 heartbeater.cc:347] Connected to a master server at 127.22.26.254:43645
I20260927 13:56:34.415930 30453 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:34.416618 30453 heartbeater.cc:511] Master 127.22.26.254:43645 requested a full tablet report, sending...
I20260927 13:56:34.417917 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 30317
I20260927 13:56:34.418159 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:56:34.418804 22635 external_mini_cluster.cc:1428] Setting key 19b92dfc313e35b4c7ce39bee1669713
I20260927 13:56:34.418963 30246 ts_manager.cc:194] Registered new tserver with Master: ebffa94065a949fdbad26729946ae3f5 (127.22.26.193:46017)
I20260927 13:56:34.420728 30246 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:44253
WARNING: no policy specified for oryx/127.22.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab.
Entry for principal oryx/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab.
I20260927 13:56:34.468248 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:43645
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab
--principal=oryx/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:34.789568 30457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:34.789894 30457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:34.790077 30457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:34.797395 30457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:34.797535 30457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:34.797695 30457 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:56:34.797842 30457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:56:34.807776 30457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab
--principal=oryx/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.26.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:34.809216 30457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:34.810685 30457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:34.824056 30465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:34.824079 30462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:34.824168 30463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:34.825368 30457 server_base.cc:1034] running on GCE node
I20260927 13:56:34.825966 30457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:34.826974 30457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:34.828261 30457 hybrid_clock.cc:648] HybridClock initialized: now 1790517394828173 us; error 62 us; skew 500 ppm
I20260927 13:56:34.828598 30457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:34 dist-test-slave-gkks krb5kdc[30201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517394, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:34.831878 30457 init.cc:377] Logged in from keytab as oryx/127.22.26.194@KRBTEST.COM (short username oryx)
I20260927 13:56:34.833284 30457 webserver.cc:460] Webserver started at http://127.22.26.194:46391/ using document root <none> and password file <none>
I20260927 13:56:34.833936 30457 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:34.834019 30457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:34.834282 30457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:34.837240 30457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "c917ff8588a5472a9c0e61c17cad7bec"
format_stamp: "Formatted at 2026-09-27 13:56:34 on dist-test-slave-gkks"
server_key: "8e24798e965f0b08f489acace9dcd023"
server_key_iv: "ada848772e660c8ac66d774e441f86db"
server_key_version: "encryptionkey@0"
I20260927 13:56:34.838066 30457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "c917ff8588a5472a9c0e61c17cad7bec"
format_stamp: "Formatted at 2026-09-27 13:56:34 on dist-test-slave-gkks"
server_key: "8e24798e965f0b08f489acace9dcd023"
server_key_iv: "ada848772e660c8ac66d774e441f86db"
server_key_version: "encryptionkey@0"
I20260927 13:56:34.843036 30457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 13:56:34.846520 30472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:34.847991 30457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:34.848179 30457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "c917ff8588a5472a9c0e61c17cad7bec"
format_stamp: "Formatted at 2026-09-27 13:56:34 on dist-test-slave-gkks"
server_key: "8e24798e965f0b08f489acace9dcd023"
server_key_iv: "ada848772e660c8ac66d774e441f86db"
server_key_version: "encryptionkey@0"
I20260927 13:56:34.848430 30457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:34.862597 30457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:34.867012 30457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:34.867374 30457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:34.870234 30457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:34.870362 30457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:34.870530 30457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:34.870723 30457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:56:34 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517394, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
I20260927 13:56:34.895889 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:34.871278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:45301 (local address 127.22.26.254:43645)
0927 13:56:34.871497 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:34.871503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:34.873039 (+ 1536us) server_negotiation.cc:407] Connection header received
0927 13:56:34.875170 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:34.875178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:34.875276 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:34.875360 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:34.878043 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.879069 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.881614 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.882280 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.882420 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:34.887193 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:34.887203 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:34.887211 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:34.887280 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:34.890894 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:34.891954 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:34.891960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:34.891965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:34.892119 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:34.892763 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:34.892772 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:34.892778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:34.893299 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:34.893507 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:34.895328 (+ 1821us) server_negotiation.cc:299] Negotiation successful
0927 13:56:34.895552 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 13:56:34.896489 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:34.871892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43645 (local address 127.22.26.194:45301)
0927 13:56:34.872837 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:34.872899 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:34.874799 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:34.875939 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:34.875954 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:34.876752 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:34.877827 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:34.877859 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.879312 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.879320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:34.881447 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:34.881460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.882534 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.882543 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:34.882877 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:34.883774 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:34.883816 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:34.886863 (+ 3047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:34.891040 (+ 4177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:34.891049 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:34.891075 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:34.891725 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:34.892261 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:34.892266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:34.892270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:34.892594 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:34.893463 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:34.893469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:34.893843 (+ 374us) client_negotiation.cc:769] Sending connection context
0927 13:56:34.894235 (+ 392us) client_negotiation.cc:240] Negotiation successful
0927 13:56:34.894693 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":405,"threads_started":1}
I20260927 13:56:34.915903 30457 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:44073
I20260927 13:56:34.915928 30591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:44073 every 8 connection(s)
I20260927 13:56:34.917392 30457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:56:34.919847 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 30457
I20260927 13:56:34.919859 30592 heartbeater.cc:347] Connected to a master server at 127.22.26.254:43645
I20260927 13:56:34.920069 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:56:34.920094 30592 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:34.920567 22635 external_mini_cluster.cc:1428] Setting key a40e53a4bc752122dea38686c3f6fa09
I20260927 13:56:34.920904 30592 heartbeater.cc:511] Master 127.22.26.254:43645 requested a full tablet report, sending...
I20260927 13:56:34.922840 30245 ts_manager.cc:194] Registered new tserver with Master: c917ff8588a5472a9c0e61c17cad7bec (127.22.26.194:44073)
I20260927 13:56:34.924180 30245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:45301
I20260927 13:56:34.927737 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:34.910688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:57127 (local address 127.22.26.254:43645)
0927 13:56:34.910883 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:34.910889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:34.912529 (+ 1640us) server_negotiation.cc:407] Connection header received
0927 13:56:34.912740 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:34.912748 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:34.912841 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:34.912954 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:34.914419 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.915748 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.917183 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.917848 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.918018 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:34.920608 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:34.920617 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:34.920622 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:34.920666 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:34.924517 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:34.925500 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:34.925510 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:34.925513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:34.925642 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:34.926041 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:34.926047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:34.926051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:34.926381 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:34.926547 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:34.927181 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 13:56:34.927398 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:56:34.927835 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:34.910155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43645 (local address 127.22.26.194:57127)
0927 13:56:34.912382 (+ 2227us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:34.912407 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:34.912583 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:34.913152 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:34.913156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:34.913856 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:34.914250 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:34.914261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.915905 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.915915 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:34.917029 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:34.917041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:34.918082 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:34.918088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:34.918386 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:34.919248 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:34.919270 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:34.920398 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:34.924743 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:34.924752 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:34.924758 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:34.925331 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:34.925763 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:34.925769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:34.925772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:34.925927 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:34.926641 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:34.926646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:34.926881 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 13:56:34.927044 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 13:56:34.927384 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1993,"thread_start_us":1826,"threads_started":1}
WARNING: no policy specified for oryx/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab.
Entry for principal oryx/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab.
I20260927 13:56:34.974850 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:43645
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab
--principal=oryx/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:35.312916 30596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:35.313279 30596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:35.313479 30596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:35.321579 30596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:35.321755 30596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:35.321918 30596 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:56:35.322060 30596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:56:35.332667 30596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab
--principal=oryx/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.26.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:35.334020 30596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:35.335484 30596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:35.348771 30601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:35.348809 30602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:35.348732 30604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:35.349856 30596 server_base.cc:1034] running on GCE node
I20260927 13:56:35.350498 30596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:35.351616 30596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:35.352890 30596 hybrid_clock.cc:648] HybridClock initialized: now 1790517395352818 us; error 39 us; skew 500 ppm
I20260927 13:56:35.353193 30596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:35 dist-test-slave-gkks krb5kdc[30201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517395, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:35.356449 30596 init.cc:377] Logged in from keytab as oryx/127.22.26.195@KRBTEST.COM (short username oryx)
I20260927 13:56:35.357969 30596 webserver.cc:460] Webserver started at http://127.22.26.195:39441/ using document root <none> and password file <none>
I20260927 13:56:35.358685 30596 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:35.358824 30596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:35.359213 30596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:35.362007 30596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "b8d93c9972cb4cb3b19900fa539c854a"
format_stamp: "Formatted at 2026-09-27 13:56:35 on dist-test-slave-gkks"
server_key: "b4a347ee60fcdcf8aab14a273fdf712a"
server_key_iv: "06f410f160972883bedd44ae29fdd7ad"
server_key_version: "encryptionkey@0"
I20260927 13:56:35.362869 30596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "b8d93c9972cb4cb3b19900fa539c854a"
format_stamp: "Formatted at 2026-09-27 13:56:35 on dist-test-slave-gkks"
server_key: "b4a347ee60fcdcf8aab14a273fdf712a"
server_key_iv: "06f410f160972883bedd44ae29fdd7ad"
server_key_version: "encryptionkey@0"
I20260927 13:56:35.368000 30596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260927 13:56:35.371479 30611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:35.372795 30596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:56:35.372973 30596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "b8d93c9972cb4cb3b19900fa539c854a"
format_stamp: "Formatted at 2026-09-27 13:56:35 on dist-test-slave-gkks"
server_key: "b4a347ee60fcdcf8aab14a273fdf712a"
server_key_iv: "06f410f160972883bedd44ae29fdd7ad"
server_key_version: "encryptionkey@0"
I20260927 13:56:35.373198 30596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:35.390240 30596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:35.394266 30596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:35.394556 30596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:35.397310 30596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:35.397444 30596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:35.397603 30596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:35.397691 30596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:56:35 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517395, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
I20260927 13:56:35.425482 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:35.397875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:42503 (local address 127.22.26.254:43645)
0927 13:56:35.398071 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:35.398077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:35.399724 (+ 1647us) server_negotiation.cc:407] Connection header received
0927 13:56:35.401777 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:35.401787 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:35.401902 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:35.402020 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:35.404358 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:35.405743 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:35.409125 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:35.409809 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:35.409967 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:35.415532 (+ 5565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:35.415539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:35.415544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:35.415584 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:35.419551 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:35.420661 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:35.420668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:35.420673 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:35.420856 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:35.421484 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:35.421490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:35.421497 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:35.421890 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:35.422025 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:35.424868 (+ 2843us) server_negotiation.cc:299] Negotiation successful
0927 13:56:35.425150 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 13:56:35.425863 30620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:35.398428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43645 (local address 127.22.26.195:42503)
0927 13:56:35.399514 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:35.399574 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:35.401478 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:35.402288 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:35.402301 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:35.403187 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:35.404067 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:35.404090 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:35.405989 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:35.405997 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:35.408896 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:35.408917 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:35.410125 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:35.410137 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:35.410413 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:35.411402 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:35.411442 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:35.415280 (+ 3838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:35.419716 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:35.419724 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:35.419738 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:35.420486 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:35.421008 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:35.421017 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:35.421026 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:35.421322 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:35.422097 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:35.422104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:35.422540 (+ 436us) client_negotiation.cc:769] Sending connection context
0927 13:56:35.422813 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 13:56:35.423335 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":563,"threads_started":1}
I20260927 13:56:35.427403 30453 heartbeater.cc:503] Master 127.22.26.254:43645 was elected leader, sending a full tablet report...
I20260927 13:56:35.453461 30688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:35.438586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43645 (local address 127.22.26.195:54769)
0927 13:56:35.439428 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:35.439450 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:35.439631 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:35.439999 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:35.440003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:35.440659 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:35.441207 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:35.441219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:35.442206 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:35.442212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:35.444095 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:35.444109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:35.444860 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:35.444866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:35.445063 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:35.445885 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:35.445904 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:35.447214 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:35.450383 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:35.450390 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:35.450396 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:35.450926 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:35.451435 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:35.451440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:35.451444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:35.451618 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:35.452391 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:35.452402 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:35.452650 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 13:56:35.452820 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 13:56:35.453125 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":429,"threads_started":1}
I20260927 13:56:35.454182 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:35.438438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:54769 (local address 127.22.26.254:43645)
0927 13:56:35.438633 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:35.438641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:35.439557 (+ 916us) server_negotiation.cc:407] Connection header received
0927 13:56:35.439784 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:35.439789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:35.439869 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:35.439975 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:35.441362 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:35.442057 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:35.444265 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:35.444665 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:35.444816 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:35.447416 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:35.447427 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:35.447431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:35.447460 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:35.450244 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:35.451159 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:35.451167 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:35.451172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:35.451335 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:35.451787 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:35.451794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:35.451800 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:35.452136 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:35.452319 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:35.453798 (+ 1479us) server_negotiation.cc:299] Negotiation successful
0927 13:56:35.453957 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:56:35.459358 30596 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:41417
I20260927 13:56:35.459398 30730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:41417 every 8 connection(s)
I20260927 13:56:35.460619 30596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:56:35.462675 30731 heartbeater.cc:347] Connected to a master server at 127.22.26.254:43645
I20260927 13:56:35.462916 30731 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:35.463601 30731 heartbeater.cc:511] Master 127.22.26.254:43645 requested a full tablet report, sending...
I20260927 13:56:35.464856 30245 ts_manager.cc:194] Registered new tserver with Master: b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195:41417)
I20260927 13:56:35.465642 30245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:42503
I20260927 13:56:35.470870 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 30596
I20260927 13:56:35.471115 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:56:35.471557 22635 external_mini_cluster.cc:1428] Setting key 9e896dc44ad6f6d2809b600d15f55b00
I20260927 13:56:35.474349 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 13:56:35 dist-test-slave-gkks krb5kdc[30201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:56:35 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:35 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:35.927958 30592 heartbeater.cc:503] Master 127.22.26.254:43645 was elected leader, sending a full tablet report...
I20260927 13:56:36.467940 30731 heartbeater.cc:503] Master 127.22.26.254:43645 was elected leader, sending a full tablet report...
W20260927 13:56:36.620293 30739 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.102s user 0.001s sys 0.014s
W20260927 13:56:36.621865 30340 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:35.512673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.22.26.254:43645)
0927 13:56:35.512894 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:35.512900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:35.513050 (+ 150us) server_negotiation.cc:407] Connection header received
0927 13:56:35.513198 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:35.513202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:35.513266 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:35.513352 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:35.514838 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:35.515645 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:35.516865 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:35.517255 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:35.517343 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:36.621492 (+1104149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45526: BlockingRecv error: recv got EOF from 127.0.0.1:45526 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 13:56:36 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
I20260927 13:56:36.644724 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:36.630751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.22.26.254:43645)
0927 13:56:36.630990 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:36.630996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:36.631312 (+ 316us) server_negotiation.cc:407] Connection header received
0927 13:56:36.631520 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:36.631525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:36.631607 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:36.631724 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:36.633052 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.633872 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.635141 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.635562 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.635710 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:36.639528 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:36.639533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:36.639536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:36.639568 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:36.642134 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:36.642765 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:36.642771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:36.642774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:36.642907 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:36.643347 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:36.643352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:36.643355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:36.643616 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:36.643932 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:36.644206 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 13:56:36.644431 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 13:56:36.648586 30245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:56:36.655478 30245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:56:36.680235 30750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:36.667477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:41417 (local address 127.0.0.1:49286)
0927 13:56:36.668592 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:36.668652 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:36.668873 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:36.670927 (+ 2054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:36.670939 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:36.670982 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:36.671568 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:36.671586 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.674286 (+ 2700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.674295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:36.677152 (+ 2857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:36.677167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.679306 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.679316 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:36.679541 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:36.679600 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 13:56:36.679703 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 13:56:36.679892 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":914,"threads_started":1}
I20260927 13:56:36.681725 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:36.667761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.22.26.195:41417)
0927 13:56:36.670201 (+ 2440us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:36.670210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:36.670239 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:56:36.670344 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:36.670353 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:36.670612 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:36.671856 (+ 1244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:36.672098 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.674080 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.677419 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.679033 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.680818 (+ 1785us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:36.681051 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:36.681135 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 13:56:36.681310 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2296,"thread_start_us":2165,"threads_started":1}
I20260927 13:56:36.685359 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:36.670001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:44073 (local address 127.0.0.1:38354)
0927 13:56:36.677671 (+ 7670us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:36.677689 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:36.677811 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:36.678642 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:36.678648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:36.678676 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:36.679002 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:36.679012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.680870 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.680875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:36.682498 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:36.682510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.684579 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.684587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:36.684761 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:36.684794 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:56:36.684885 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 13:56:36.685001 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7554,"mutex_wait_us":5,"thread_start_us":7384,"threads_started":1}
I20260927 13:56:36.685789 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:36.668450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.22.26.193:46017)
0927 13:56:36.676838 (+ 8388us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:36.676847 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:36.676875 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:56:36.677699 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:36.677710 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:36.677959 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:36.678178 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:36.678768 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.680175 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.682974 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.684230 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.684377 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:36.684595 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:36.685308 (+ 713us) server_negotiation.cc:299] Negotiation successful
0927 13:56:36.685471 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8219,"thread_start_us":8158,"threads_started":1}
I20260927 13:56:36.685832 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:36.667915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:46017 (local address 127.0.0.1:53106)
0927 13:56:36.669472 (+ 1557us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:36.669498 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:36.669651 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:36.678162 (+ 8511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:36.678167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:36.678196 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:36.678582 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:36.678592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.680313 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.680321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:36.682792 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:36.682809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.684762 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.684769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:36.685090 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:36.685122 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 13:56:36.685263 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 13:56:36.685406 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"mutex_wait_us":48,"thread_start_us":1226,"threads_started":1}
I20260927 13:56:36.686110 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:36.670539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.22.26.194:44073)
0927 13:56:36.677392 (+ 6853us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:36.677405 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:36.677932 (+ 527us) server_negotiation.cc:407] Connection header received
0927 13:56:36.678034 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:36.678041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:36.678256 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:36.678430 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:36.679235 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.680701 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.682654 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.684119 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.684249 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:36.684431 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:36.685658 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0927 13:56:36.685836 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6626,"thread_start_us":6456,"threads_started":1}
I20260927 13:56:36.685858 30661 tablet_service.cc:1467] Processing CreateTablet for tablet 57678e70bcd0497996af33e2770c7ad8 (DEFAULT_TABLE table=test-table [id=bd397536eae146e4b4c3bf8757040a26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:36.687425 30661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57678e70bcd0497996af33e2770c7ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:36.689270 30522 tablet_service.cc:1467] Processing CreateTablet for tablet 57678e70bcd0497996af33e2770c7ad8 (DEFAULT_TABLE table=test-table [id=bd397536eae146e4b4c3bf8757040a26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:36.689838 30383 tablet_service.cc:1467] Processing CreateTablet for tablet 57678e70bcd0497996af33e2770c7ad8 (DEFAULT_TABLE table=test-table [id=bd397536eae146e4b4c3bf8757040a26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:36.690932 30522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57678e70bcd0497996af33e2770c7ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:36.691140 30383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57678e70bcd0497996af33e2770c7ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:36.702495 30756 tablet_bootstrap.cc:492] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a: Bootstrap starting.
I20260927 13:56:36.704159 30757 tablet_bootstrap.cc:492] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec: Bootstrap starting.
I20260927 13:56:36.705647 30758 tablet_bootstrap.cc:492] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5: Bootstrap starting.
I20260927 13:56:36.707754 30757 tablet_bootstrap.cc:654] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:36.708477 30756 tablet_bootstrap.cc:654] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:36.709095 30757 log.cc:824] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:36.709559 30758 tablet_bootstrap.cc:654] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:36.709820 30756 log.cc:824] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:36.711067 30758 log.cc:824] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:36.712095 30757 tablet_bootstrap.cc:492] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec: No bootstrap required, opened a new log
I20260927 13:56:36.712363 30757 ts_tablet_manager.cc:1397] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 13:56:36.713488 30756 tablet_bootstrap.cc:492] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a: No bootstrap required, opened a new log
I20260927 13:56:36.713835 30756 ts_tablet_manager.cc:1397] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260927 13:56:36.719959 30757 raft_consensus.cc:348] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } }
I20260927 13:56:36.720314 30757 raft_consensus.cc:374] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:36.720441 30757 raft_consensus.cc:729] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c917ff8588a5472a9c0e61c17cad7bec, State: Initialized, Role: FOLLOWER
I20260927 13:56:36.721115 30757 consensus_queue.cc:260] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } }
I20260927 13:56:36.722081 30756 raft_consensus.cc:348] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } }
I20260927 13:56:36.722640 30756 raft_consensus.cc:374] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:36.722785 30756 raft_consensus.cc:729] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d93c9972cb4cb3b19900fa539c854a, State: Initialized, Role: FOLLOWER
I20260927 13:56:36.722862 30758 tablet_bootstrap.cc:492] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5: No bootstrap required, opened a new log
I20260927 13:56:36.723009 30757 ts_tablet_manager.cc:1428] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 13:56:36.723259 30758 ts_tablet_manager.cc:1397] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.003s
I20260927 13:56:36.723750 30756 consensus_queue.cc:260] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } }
I20260927 13:56:36.726805 30756 ts_tablet_manager.cc:1428] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260927 13:56:36.732525 30758 raft_consensus.cc:348] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } }
I20260927 13:56:36.732923 30758 raft_consensus.cc:374] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:36.733033 30758 raft_consensus.cc:729] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebffa94065a949fdbad26729946ae3f5, State: Initialized, Role: FOLLOWER
I20260927 13:56:36.733633 30758 consensus_queue.cc:260] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } }
I20260927 13:56:36.735213 30758 ts_tablet_manager.cc:1428] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260927 13:56:36.781553 30762 raft_consensus.cc:482] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:56:36.781826 30762 raft_consensus.cc:504] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } }
I20260927 13:56:36.783643 30762 leader_election.cc:288] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195:41417), ebffa94065a949fdbad26729946ae3f5 (127.22.26.193:46017)
I20260927 13:56:36.790949 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:36.784284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:50689 (local address 127.22.26.193:46017)
0927 13:56:36.784529 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:36.784539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:36.785262 (+ 723us) server_negotiation.cc:407] Connection header received
0927 13:56:36.785492 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:36.785504 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:36.785572 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:36.785672 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:36.786283 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.787008 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.788995 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.789964 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.790118 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:36.790193 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:36.790559 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 13:56:36.790687 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:56:36.791062 30766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:36.783950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:46017 (local address 127.22.26.194:50689)
0927 13:56:36.785130 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:36.785171 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:36.785337 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:36.785700 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:36.785704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:36.785728 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:36.786127 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:36.786139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.787164 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.787171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:36.788832 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:36.788844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.790211 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.790218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:36.790363 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:36.790393 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:56:36.790500 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 13:56:36.790645 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":77,"spinlock_wait_cycles":3328,"thread_start_us":966,"threads_started":1}
I20260927 13:56:36.791389 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:36.784025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:33123 (local address 127.22.26.195:41417)
0927 13:56:36.784293 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:36.784303 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:36.785351 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 13:56:36.785555 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:36.785560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:36.785629 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:36.785738 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:36.786346 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.787023 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.788971 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.789828 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.789972 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:36.790052 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:36.790993 (+ 941us) server_negotiation.cc:299] Negotiation successful
0927 13:56:36.791157 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:56:36.791843 30765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:36.783997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:41417 (local address 127.22.26.194:33123)
0927 13:56:36.785234 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:36.785248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:36.785398 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:36.785763 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:36.785767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:36.785795 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:36.786178 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:36.786193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.787215 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.787223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:36.788817 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:36.788829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.790626 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.790631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:36.790805 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:36.790838 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:56:36.790937 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 13:56:36.791075 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1105,"mutex_wait_us":25,"thread_start_us":532,"threads_started":1}
I20260927 13:56:36.792382 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57678e70bcd0497996af33e2770c7ad8" candidate_uuid: "c917ff8588a5472a9c0e61c17cad7bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebffa94065a949fdbad26729946ae3f5" is_pre_election: true
I20260927 13:56:36.792845 30685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57678e70bcd0497996af33e2770c7ad8" candidate_uuid: "c917ff8588a5472a9c0e61c17cad7bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d93c9972cb4cb3b19900fa539c854a" is_pre_election: true
I20260927 13:56:36.792980 30407 raft_consensus.cc:2454] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c917ff8588a5472a9c0e61c17cad7bec in term 0.
I20260927 13:56:36.793392 30685 raft_consensus.cc:2454] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c917ff8588a5472a9c0e61c17cad7bec in term 0.
I20260927 13:56:36.793658 30475 leader_election.cc:302] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c917ff8588a5472a9c0e61c17cad7bec, ebffa94065a949fdbad26729946ae3f5; no voters:
I20260927 13:56:36.794094 30762 raft_consensus.cc:2781] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:56:36.794214 30762 raft_consensus.cc:482] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:56:36.794371 30762 raft_consensus.cc:3037] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:36.796626 30762 raft_consensus.cc:504] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } }
I20260927 13:56:36.797310 30762 leader_election.cc:288] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [CANDIDATE]: Term 1 election: Requested vote from peers b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195:41417), ebffa94065a949fdbad26729946ae3f5 (127.22.26.193:46017)
I20260927 13:56:36.797710 30685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57678e70bcd0497996af33e2770c7ad8" candidate_uuid: "c917ff8588a5472a9c0e61c17cad7bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d93c9972cb4cb3b19900fa539c854a"
I20260927 13:56:36.797828 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57678e70bcd0497996af33e2770c7ad8" candidate_uuid: "c917ff8588a5472a9c0e61c17cad7bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebffa94065a949fdbad26729946ae3f5"
I20260927 13:56:36.797910 30685 raft_consensus.cc:3037] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:36.798031 30407 raft_consensus.cc:3037] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:36.801113 30407 raft_consensus.cc:2454] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c917ff8588a5472a9c0e61c17cad7bec in term 1.
I20260927 13:56:36.801113 30685 raft_consensus.cc:2454] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c917ff8588a5472a9c0e61c17cad7bec in term 1.
I20260927 13:56:36.801494 30476 leader_election.cc:302] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d93c9972cb4cb3b19900fa539c854a, c917ff8588a5472a9c0e61c17cad7bec; no voters:
I20260927 13:56:36.801800 30762 raft_consensus.cc:2781] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:36.802224 30762 raft_consensus.cc:686] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [term 1 LEADER]: Becoming Leader. State: Replica: c917ff8588a5472a9c0e61c17cad7bec, State: Running, Role: LEADER
I20260927 13:56:36.802665 30762 consensus_queue.cc:237] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } }
I20260927 13:56:36.806846 30245 catalog_manager.cc:5495] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec reported cstate change: term changed from 0 to 1, leader changed from <none> to c917ff8588a5472a9c0e61c17cad7bec (127.22.26.194). New cstate: current_term: 1 leader_uuid: "c917ff8588a5472a9c0e61c17cad7bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } health_report { overall_health: HEALTHY } } }
I20260927 13:56:36.832628 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:36.827432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38364 (local address 127.22.26.194:44073)
0927 13:56:36.827700 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:36.827705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:36.827722 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:56:36.827899 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:36.827902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:36.827988 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:36.828107 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:56:36.828422 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.829102 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.830418 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:36.830910 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:36.831004 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:36.831493 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:56:36.831998 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:56:36.832103 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:36.832230 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 13:56:36.832292 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:56:36.840456 30407 raft_consensus.cc:1261] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Refusing update from remote peer c917ff8588a5472a9c0e61c17cad7bec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:36.840488 30685 raft_consensus.cc:1261] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Refusing update from remote peer c917ff8588a5472a9c0e61c17cad7bec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:36.841426 30767 consensus_queue.cc:1035] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:36.841773 30762 consensus_queue.cc:1035] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:36.851114 30770 mvcc.cc:205] Tried to move back new op lower bound from 7333959257448722432 to 7333959257307418624. Current Snapshot: MvccSnapshot[applied={T|T < 7333959257448722432}]
I20260927 13:56:36.864907 30772 mvcc.cc:205] Tried to move back new op lower bound from 7333959257448722432 to 7333959257307418624. Current Snapshot: MvccSnapshot[applied={T|T < 7333959257448722432}]
I20260927 13:56:36.865998 30771 mvcc.cc:205] Tried to move back new op lower bound from 7333959257448722432 to 7333959257307418624. Current Snapshot: MvccSnapshot[applied={T|T < 7333959257448722432 or (T in {7333959257448722432})}]
I20260927 13:56:36.866053 30245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:36.866295 30245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:36.870766 30245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100: Sending DeleteTablet for 3 replicas of tablet 57678e70bcd0497996af33e2770c7ad8
I20260927 13:56:36.871956 30383 tablet_service.cc:1514] Processing DeleteTablet for tablet 57678e70bcd0497996af33e2770c7ad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:36 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:53106
I20260927 13:56:36.871994 30661 tablet_service.cc:1514] Processing DeleteTablet for tablet 57678e70bcd0497996af33e2770c7ad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:36 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:49286
I20260927 13:56:36.872041 30660 tablet_service.cc:1514] Processing DeleteTablet for tablet 57678e70bcd0497996af33e2770c7ad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:36 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:49286
I20260927 13:56:36.872502 30522 tablet_service.cc:1514] Processing DeleteTablet for tablet 57678e70bcd0497996af33e2770c7ad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:36 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:38354
I20260927 13:56:36.872941 30779 tablet_replica.cc:330] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5: stopping tablet replica
I20260927 13:56:36.873342 30779 raft_consensus.cc:2229] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:36.873811 30779 raft_consensus.cc:2258] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:36.875950 30779 ts_tablet_manager.cc:1905] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:36.877550 30246 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:56:36.878383 30246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:56:36.879665 30780 tablet_replica.cc:330] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a: stopping tablet replica
I20260927 13:56:36.880079 30780 raft_consensus.cc:2229] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:36.880504 30780 raft_consensus.cc:2258] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:36.884907 30779 ts_tablet_manager.cc:1918] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:36.885059 30779 log.cc:1196] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/57678e70bcd0497996af33e2770c7ad8
I20260927 13:56:36.885591 30779 ts_tablet_manager.cc:1939] T 57678e70bcd0497996af33e2770c7ad8 P ebffa94065a949fdbad26729946ae3f5: Deleting consensus metadata
I20260927 13:56:36.887032 30383 tablet_service.cc:1467] Processing CreateTablet for tablet c5e4fdff7f9b406580e557b365b9dbc6 (DEFAULT_TABLE table=test-table [id=c60eaece608d4044a671bc1aa054825f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:36.887578 30383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e4fdff7f9b406580e557b365b9dbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:36.887570 30781 tablet_replica.cc:330] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec: stopping tablet replica
I20260927 13:56:36.887769 30522 tablet_service.cc:1467] Processing CreateTablet for tablet c5e4fdff7f9b406580e557b365b9dbc6 (DEFAULT_TABLE table=test-table [id=c60eaece608d4044a671bc1aa054825f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:36.887804 30660 tablet_service.cc:1467] Processing CreateTablet for tablet c5e4fdff7f9b406580e557b365b9dbc6 (DEFAULT_TABLE table=test-table [id=c60eaece608d4044a671bc1aa054825f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:36.888036 30781 raft_consensus.cc:2229] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:56:36.888316 30660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e4fdff7f9b406580e557b365b9dbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:36.888688 30781 raft_consensus.cc:2258] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:36.889447 30232 catalog_manager.cc:4841] TS ebffa94065a949fdbad26729946ae3f5 (127.22.26.193:46017): tablet 57678e70bcd0497996af33e2770c7ad8 (table test-table [id=bd397536eae146e4b4c3bf8757040a26]) successfully deleted
I20260927 13:56:36.889987 30522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e4fdff7f9b406580e557b365b9dbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:36.890627 30780 ts_tablet_manager.cc:1905] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:36.894035 30758 tablet_bootstrap.cc:492] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5: Bootstrap starting.
I20260927 13:56:36.895669 30758 tablet_bootstrap.cc:654] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:36.895892 30781 ts_tablet_manager.cc:1905] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:36.896526 30757 tablet_bootstrap.cc:492] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec: Bootstrap starting.
I20260927 13:56:36.897921 30757 tablet_bootstrap.cc:654] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:36.900302 30757 tablet_bootstrap.cc:492] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec: No bootstrap required, opened a new log
I20260927 13:56:36.900439 30757 ts_tablet_manager.cc:1397] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260927 13:56:36.901238 30757 raft_consensus.cc:348] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:36.901443 30757 raft_consensus.cc:374] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:36.901510 30757 raft_consensus.cc:729] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c917ff8588a5472a9c0e61c17cad7bec, State: Initialized, Role: FOLLOWER
I20260927 13:56:36.901752 30757 consensus_queue.cc:260] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:36.902446 30757 ts_tablet_manager.cc:1428] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 13:56:36.907382 30780 ts_tablet_manager.cc:1918] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:36.907544 30780 log.cc:1196] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/57678e70bcd0497996af33e2770c7ad8
I20260927 13:56:36.907701 30758 tablet_bootstrap.cc:492] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5: No bootstrap required, opened a new log
I20260927 13:56:36.907873 30758 ts_tablet_manager.cc:1397] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260927 13:56:36.908200 30780 ts_tablet_manager.cc:1939] T 57678e70bcd0497996af33e2770c7ad8 P b8d93c9972cb4cb3b19900fa539c854a: Deleting consensus metadata
I20260927 13:56:36.908742 30758 raft_consensus.cc:348] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:36.908953 30758 raft_consensus.cc:374] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:36.909055 30758 raft_consensus.cc:729] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebffa94065a949fdbad26729946ae3f5, State: Initialized, Role: FOLLOWER
I20260927 13:56:36.909310 30758 consensus_queue.cc:260] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:36.909986 30758 ts_tablet_manager.cc:1428] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 13:56:36.910274 30233 catalog_manager.cc:4841] TS b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195:41417): tablet 57678e70bcd0497996af33e2770c7ad8 (table test-table [id=bd397536eae146e4b4c3bf8757040a26]) successfully deleted
I20260927 13:56:36.911072 30756 tablet_bootstrap.cc:492] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a: Bootstrap starting.
W20260927 13:56:36.911757 30233 catalog_manager.cc:4808] TS b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195:41417): delete failed for tablet 57678e70bcd0497996af33e2770c7ad8 because the tablet was not found. No further retry: Not found: Tablet not found: 57678e70bcd0497996af33e2770c7ad8
I20260927 13:56:36.912711 30756 tablet_bootstrap.cc:654] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:36.915107 30756 tablet_bootstrap.cc:492] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a: No bootstrap required, opened a new log
I20260927 13:56:36.915254 30756 ts_tablet_manager.cc:1397] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260927 13:56:36.916020 30756 raft_consensus.cc:348] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:36.916218 30756 raft_consensus.cc:374] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:36.916321 30756 raft_consensus.cc:729] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d93c9972cb4cb3b19900fa539c854a, State: Initialized, Role: FOLLOWER
I20260927 13:56:36.916827 30756 consensus_queue.cc:260] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
W20260927 13:56:36.918123 30454 tablet.cc:2369] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:56:36.920199 30781 ts_tablet_manager.cc:1918] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:36.920351 30781 log.cc:1196] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/57678e70bcd0497996af33e2770c7ad8
I20260927 13:56:36.920856 30756 ts_tablet_manager.cc:1428] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
W20260927 13:56:36.921077 30593 tablet.cc:2369] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:56:36.921104 30781 ts_tablet_manager.cc:1939] T 57678e70bcd0497996af33e2770c7ad8 P c917ff8588a5472a9c0e61c17cad7bec: Deleting consensus metadata
I20260927 13:56:36.922958 30233 catalog_manager.cc:4841] TS c917ff8588a5472a9c0e61c17cad7bec (127.22.26.194:44073): tablet 57678e70bcd0497996af33e2770c7ad8 (table test-table [id=bd397536eae146e4b4c3bf8757040a26]) successfully deleted
W20260927 13:56:36.963177 30732 tablet.cc:2369] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:56:37.097129 30764 raft_consensus.cc:482] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:56:37.097335 30764 raft_consensus.cc:504] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.099143 30764 leader_election.cc:288] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c917ff8588a5472a9c0e61c17cad7bec (127.22.26.194:44073), b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195:41417)
I20260927 13:56:37.108173 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.100607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:40089 (local address 127.22.26.195:41417)
0927 13:56:37.100790 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:37.100796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:37.101418 (+ 622us) server_negotiation.cc:407] Connection header received
0927 13:56:37.101589 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:37.101594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:37.101680 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:37.101768 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:37.102239 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.102932 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.105864 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.107132 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.107320 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.107397 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:37.107720 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 13:56:37.107882 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 13:56:37.108316 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.100428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:41417 (local address 127.22.26.193:40089)
0927 13:56:37.101263 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:37.101285 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:37.101435 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:37.101797 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:37.101801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:37.101824 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:37.102078 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.102085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.103065 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.103074 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.105690 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.105704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.107356 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.107362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.107535 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.107570 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:56:37.107671 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 13:56:37.107846 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":133,"thread_start_us":525,"threads_started":1}
I20260927 13:56:37.109447 30685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e4fdff7f9b406580e557b365b9dbc6" candidate_uuid: "ebffa94065a949fdbad26729946ae3f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d93c9972cb4cb3b19900fa539c854a" is_pre_election: true
I20260927 13:56:37.109714 30685 raft_consensus.cc:2454] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebffa94065a949fdbad26729946ae3f5 in term 0.
I20260927 13:56:37.109834 30785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.099541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:44073 (local address 127.22.26.193:56319)
0927 13:56:37.100282 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:37.100304 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:37.100502 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:37.101154 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:37.101157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:37.101178 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:37.101587 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.101600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.103170 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.103176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.105689 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.105704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.109008 (+ 3304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.109014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.109188 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.109241 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 13:56:37.109341 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 13:56:37.109465 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":182,"thread_start_us":459,"threads_started":1}
I20260927 13:56:37.110327 30336 leader_election.cc:302] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d93c9972cb4cb3b19900fa539c854a, ebffa94065a949fdbad26729946ae3f5; no voters:
I20260927 13:56:37.110533 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.099578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:56319 (local address 127.22.26.194:44073)
0927 13:56:37.099814 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:37.099823 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:37.100382 (+ 559us) server_negotiation.cc:407] Connection header received
0927 13:56:37.100736 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:37.100744 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:37.100851 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:37.100931 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:37.101788 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.102982 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.107245 (+ 4263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.108548 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.108744 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.108823 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:37.109391 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 13:56:37.109578 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 13:56:37.110746 30764 raft_consensus.cc:2781] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:56:37.110884 30764 raft_consensus.cc:482] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:56:37.110980 30764 raft_consensus.cc:3037] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:37.111351 30542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e4fdff7f9b406580e557b365b9dbc6" candidate_uuid: "ebffa94065a949fdbad26729946ae3f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c917ff8588a5472a9c0e61c17cad7bec" is_pre_election: true
I20260927 13:56:37.111824 30542 raft_consensus.cc:2454] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebffa94065a949fdbad26729946ae3f5 in term 0.
I20260927 13:56:37.113588 30764 raft_consensus.cc:504] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.114197 30764 leader_election.cc:288] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [CANDIDATE]: Term 1 election: Requested vote from peers c917ff8588a5472a9c0e61c17cad7bec (127.22.26.194:44073), b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195:41417)
I20260927 13:56:37.114830 30685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e4fdff7f9b406580e557b365b9dbc6" candidate_uuid: "ebffa94065a949fdbad26729946ae3f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d93c9972cb4cb3b19900fa539c854a"
I20260927 13:56:37.114849 30542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e4fdff7f9b406580e557b365b9dbc6" candidate_uuid: "ebffa94065a949fdbad26729946ae3f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c917ff8588a5472a9c0e61c17cad7bec"
I20260927 13:56:37.115025 30685 raft_consensus.cc:3037] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:37.115168 30542 raft_consensus.cc:3037] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:37.117046 30685 raft_consensus.cc:2454] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebffa94065a949fdbad26729946ae3f5 in term 1.
I20260927 13:56:37.117413 30336 leader_election.cc:302] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d93c9972cb4cb3b19900fa539c854a, ebffa94065a949fdbad26729946ae3f5; no voters:
I20260927 13:56:37.117700 30764 raft_consensus.cc:2781] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:37.118216 30764 raft_consensus.cc:686] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [term 1 LEADER]: Becoming Leader. State: Replica: ebffa94065a949fdbad26729946ae3f5, State: Running, Role: LEADER
I20260927 13:56:37.118603 30764 consensus_queue.cc:237] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.118922 30542 raft_consensus.cc:2454] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebffa94065a949fdbad26729946ae3f5 in term 1.
I20260927 13:56:37.122182 30245 catalog_manager.cc:5495] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebffa94065a949fdbad26729946ae3f5 (127.22.26.193). New cstate: current_term: 1 leader_uuid: "ebffa94065a949fdbad26729946ae3f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } health_report { overall_health: UNKNOWN } } }
Sep 27 13:56:37 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517393, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.254@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
I20260927 13:56:37.173233 30794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.157548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43645 (local address 127.0.0.1:45538)
0927 13:56:37.158408 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:37.158432 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:37.158604 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:37.159172 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:37.159177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:37.160181 (+ 1004us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:37.160553 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.160567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.161711 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.161720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.163370 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.163385 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.164128 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.164136 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.164306 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.165009 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:37.165034 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:37.167673 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:37.170663 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:37.170676 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:37.170684 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:37.171154 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:37.171494 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:37.171500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:37.171504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:37.171653 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:37.172176 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:37.172183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:37.172354 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 13:56:37.172509 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 13:56:37.172801 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":412,"threads_started":1}
I20260927 13:56:37.173352 30795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.157681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.22.26.254:43645)
0927 13:56:37.158796 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:37.158800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:37.158823 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:56:37.158870 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:37.158873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:37.158966 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:37.159069 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:37.160731 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.161540 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.163518 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.163901 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.163991 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.167874 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:37.167879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:37.167882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:37.167907 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:37.170513 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:37.171275 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:37.171279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:37.171281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:37.171388 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:37.171765 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:37.171770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:37.171777 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:37.172040 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:37.172199 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:37.172602 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 13:56:37.172940 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":1014,"thread_start_us":782,"threads_started":1}
I20260927 13:56:37.178817 30245 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:45538:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 13:56:37.179946 30245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:56:37.187906 30383 tablet_service.cc:1467] Processing CreateTablet for tablet e61e356d91c645d3ac3627e53a917aa9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0651c801d3614505a01cde3aa9fc8292]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:56:37.188552 30383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61e356d91c645d3ac3627e53a917aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:37.188346 30522 tablet_service.cc:1467] Processing CreateTablet for tablet e61e356d91c645d3ac3627e53a917aa9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0651c801d3614505a01cde3aa9fc8292]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:56:37.188922 30522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61e356d91c645d3ac3627e53a917aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:37.191848 30660 tablet_service.cc:1467] Processing CreateTablet for tablet e61e356d91c645d3ac3627e53a917aa9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0651c801d3614505a01cde3aa9fc8292]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:56:37.192374 30660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61e356d91c645d3ac3627e53a917aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:37.194880 30758 tablet_bootstrap.cc:492] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5: Bootstrap starting.
I20260927 13:56:37.196067 30757 tablet_bootstrap.cc:492] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec: Bootstrap starting.
I20260927 13:56:37.196511 30758 tablet_bootstrap.cc:654] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:37.197947 30757 tablet_bootstrap.cc:654] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:37.198546 30756 tablet_bootstrap.cc:492] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a: Bootstrap starting.
I20260927 13:56:37.199138 30758 tablet_bootstrap.cc:492] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5: No bootstrap required, opened a new log
I20260927 13:56:37.199352 30758 ts_tablet_manager.cc:1397] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 13:56:37.200254 30756 tablet_bootstrap.cc:654] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:37.200357 30758 raft_consensus.cc:348] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.200703 30758 raft_consensus.cc:374] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:37.200802 30758 raft_consensus.cc:729] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebffa94065a949fdbad26729946ae3f5, State: Initialized, Role: FOLLOWER
I20260927 13:56:37.202091 30758 consensus_queue.cc:260] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.202765 30764 tablet_replica.cc:439] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } }
I20260927 13:56:37.203200 30764 tablet_replica.cc:442] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:56:37.205015 30758 ts_tablet_manager.cc:1428] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260927 13:56:37.207489 30756 tablet_bootstrap.cc:492] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a: No bootstrap required, opened a new log
I20260927 13:56:37.207667 30756 ts_tablet_manager.cc:1397] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 13:56:37.208611 30756 raft_consensus.cc:348] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.208853 30756 raft_consensus.cc:374] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:37.208933 30756 raft_consensus.cc:729] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d93c9972cb4cb3b19900fa539c854a, State: Initialized, Role: FOLLOWER
I20260927 13:56:37.209218 30756 consensus_queue.cc:260] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.209843 30757 tablet_bootstrap.cc:492] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec: No bootstrap required, opened a new log
I20260927 13:56:37.209667 30763 tablet_replica.cc:439] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } }
I20260927 13:56:37.210018 30763 tablet_replica.cc:442] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:56:37.210114 30757 ts_tablet_manager.cc:1397] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20260927 13:56:37.210963 30756 ts_tablet_manager.cc:1428] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 13:56:37.210860 30757 raft_consensus.cc:348] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.211187 30757 raft_consensus.cc:374] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:37.211335 30757 raft_consensus.cc:729] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c917ff8588a5472a9c0e61c17cad7bec, State: Initialized, Role: FOLLOWER
I20260927 13:56:37.211686 30757 consensus_queue.cc:260] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.212212 30767 tablet_replica.cc:439] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } }
I20260927 13:56:37.212595 30767 tablet_replica.cc:442] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:56:37.213217 30757 ts_tablet_manager.cc:1428] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 13:56:37.233615 30763 raft_consensus.cc:482] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:56:37.233956 30763 raft_consensus.cc:504] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.235576 30763 leader_election.cc:288] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebffa94065a949fdbad26729946ae3f5 (127.22.26.193:46017), c917ff8588a5472a9c0e61c17cad7bec (127.22.26.194:44073)
I20260927 13:56:37.243796 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.235989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:56121 (local address 127.22.26.193:46017)
0927 13:56:37.236260 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:37.236270 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:37.237351 (+ 1081us) server_negotiation.cc:407] Connection header received
0927 13:56:37.237708 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:37.237717 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:37.237806 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:37.237866 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:37.238774 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.239767 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.241858 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.242637 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.242845 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.242925 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:37.243351 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 13:56:37.243504 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:56:37.243855 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.235727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:46017 (local address 127.22.26.195:56121)
0927 13:56:37.237226 (+ 1499us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:37.237250 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:37.237492 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:37.238065 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:37.238070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:37.238090 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:37.238551 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.238576 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.239945 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.239950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.241685 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.241696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.242872 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.242879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.243161 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.243197 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 13:56:37.243296 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 13:56:37.243436 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1328,"mutex_wait_us":67,"spinlock_wait_cycles":4992,"thread_start_us":2024,"threads_started":1}
I20260927 13:56:37.244697 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61e356d91c645d3ac3627e53a917aa9" candidate_uuid: "b8d93c9972cb4cb3b19900fa539c854a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebffa94065a949fdbad26729946ae3f5" is_pre_election: true
I20260927 13:56:37.244959 30407 raft_consensus.cc:2454] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8d93c9972cb4cb3b19900fa539c854a in term 0.
I20260927 13:56:37.245610 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.236328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:58047 (local address 127.22.26.194:44073)
0927 13:56:37.236500 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:37.236506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:37.238192 (+ 1686us) server_negotiation.cc:407] Connection header received
0927 13:56:37.238381 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:37.238386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:37.238492 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:37.238597 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:56:37.239456 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.240608 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.243207 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.244528 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.244722 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.244789 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:37.245183 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 13:56:37.245330 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 13:56:37.245656 30799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.235912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:44073 (local address 127.22.26.195:58047)
0927 13:56:37.238046 (+ 2134us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:37.238070 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:37.238206 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:37.238668 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:37.238673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:37.238701 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:56:37.239239 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.239257 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.240820 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.240824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.243037 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.243052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.244744 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.244750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.244952 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.244989 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 13:56:37.245102 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 13:56:37.245222 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2013,"mutex_wait_us":1,"thread_start_us":1536,"threads_started":1}
I20260927 13:56:37.245553 30614 leader_election.cc:302] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d93c9972cb4cb3b19900fa539c854a, ebffa94065a949fdbad26729946ae3f5; no voters:
I20260927 13:56:37.246170 30763 raft_consensus.cc:2781] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:56:37.246299 30763 raft_consensus.cc:482] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:56:37.246407 30763 raft_consensus.cc:3037] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:37.246516 30542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61e356d91c645d3ac3627e53a917aa9" candidate_uuid: "b8d93c9972cb4cb3b19900fa539c854a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c917ff8588a5472a9c0e61c17cad7bec" is_pre_election: true
I20260927 13:56:37.246774 30542 raft_consensus.cc:2454] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8d93c9972cb4cb3b19900fa539c854a in term 0.
I20260927 13:56:37.248950 30763 raft_consensus.cc:504] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.249651 30763 leader_election.cc:288] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [CANDIDATE]: Term 1 election: Requested vote from peers ebffa94065a949fdbad26729946ae3f5 (127.22.26.193:46017), c917ff8588a5472a9c0e61c17cad7bec (127.22.26.194:44073)
I20260927 13:56:37.250121 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61e356d91c645d3ac3627e53a917aa9" candidate_uuid: "b8d93c9972cb4cb3b19900fa539c854a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebffa94065a949fdbad26729946ae3f5"
I20260927 13:56:37.250151 30542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61e356d91c645d3ac3627e53a917aa9" candidate_uuid: "b8d93c9972cb4cb3b19900fa539c854a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c917ff8588a5472a9c0e61c17cad7bec"
I20260927 13:56:37.250353 30407 raft_consensus.cc:3037] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:37.250373 30542 raft_consensus.cc:3037] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:37.252849 30542 raft_consensus.cc:2454] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d93c9972cb4cb3b19900fa539c854a in term 1.
I20260927 13:56:37.253259 30615 leader_election.cc:302] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d93c9972cb4cb3b19900fa539c854a, c917ff8588a5472a9c0e61c17cad7bec; no voters:
I20260927 13:56:37.253396 30407 raft_consensus.cc:2454] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d93c9972cb4cb3b19900fa539c854a in term 1.
I20260927 13:56:37.253525 30763 raft_consensus.cc:2781] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:37.254066 30763 raft_consensus.cc:686] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [term 1 LEADER]: Becoming Leader. State: Replica: b8d93c9972cb4cb3b19900fa539c854a, State: Running, Role: LEADER
I20260927 13:56:37.254477 30763 consensus_queue.cc:237] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.256685 30801 tablet_replica.cc:439] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a: TxnStatusTablet state changed. Reason: New leader b8d93c9972cb4cb3b19900fa539c854a. Latest consensus state: current_term: 1 leader_uuid: "b8d93c9972cb4cb3b19900fa539c854a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } }
I20260927 13:56:37.257131 30801 tablet_replica.cc:442] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:56:37.257915 30803 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 13:56:37.258167 30803 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 13:56:37.258751 30246 catalog_manager.cc:5495] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a reported cstate change: term changed from 0 to 1, leader changed from <none> to b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195). New cstate: current_term: 1 leader_uuid: "b8d93c9972cb4cb3b19900fa539c854a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } health_report { overall_health: HEALTHY } } }
I20260927 13:56:37.294747 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.287450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.22.26.195:41417)
0927 13:56:37.287645 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:37.287651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:37.287809 (+ 158us) server_negotiation.cc:407] Connection header received
0927 13:56:37.288083 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:37.288089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:37.288221 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:37.288395 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:56:37.288817 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.290060 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.292051 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.292715 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.292848 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.293391 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:56:37.294049 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:56:37.294183 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:37.294330 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 13:56:37.294415 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:56:37.294771 30794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.287378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:41417 (local address 127.0.0.1:49292)
0927 13:56:37.287709 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:37.287729 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:37.287873 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:37.288435 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:37.288440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:37.288453 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:56:37.288664 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.288676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.290215 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.290221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.291892 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.291908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.292899 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.292905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.293121 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.293190 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:56:37.294177 (+ 987us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:56:37.294200 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 13:56:37.294294 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 13:56:37.294455 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260927 13:56:37.299484 30407 raft_consensus.cc:1261] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Refusing update from remote peer b8d93c9972cb4cb3b19900fa539c854a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:37.299505 30542 raft_consensus.cc:1261] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Refusing update from remote peer b8d93c9972cb4cb3b19900fa539c854a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:37.300276 30801 consensus_queue.cc:1035] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:37.300947 30763 consensus_queue.cc:1035] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:37.311383 30764 tablet_replica.cc:439] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5: TxnStatusTablet state changed. Reason: New leader b8d93c9972cb4cb3b19900fa539c854a. Latest consensus state: current_term: 1 leader_uuid: "b8d93c9972cb4cb3b19900fa539c854a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } }
I20260927 13:56:37.311710 30764 tablet_replica.cc:442] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:56:37.312413 30767 tablet_replica.cc:439] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec: TxnStatusTablet state changed. Reason: New leader b8d93c9972cb4cb3b19900fa539c854a. Latest consensus state: current_term: 1 leader_uuid: "b8d93c9972cb4cb3b19900fa539c854a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } }
I20260927 13:56:37.312726 30767 tablet_replica.cc:442] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:56:37.315019 30763 tablet_replica.cc:439] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8d93c9972cb4cb3b19900fa539c854a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } }
I20260927 13:56:37.315321 30763 tablet_replica.cc:442] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:56:37.315882 30763 tablet_replica.cc:439] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8d93c9972cb4cb3b19900fa539c854a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } }
I20260927 13:56:37.316097 30763 tablet_replica.cc:442] T e61e356d91c645d3ac3627e53a917aa9 P b8d93c9972cb4cb3b19900fa539c854a: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:56:37.320106 30767 tablet_replica.cc:439] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8d93c9972cb4cb3b19900fa539c854a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } }
I20260927 13:56:37.320358 30767 tablet_replica.cc:442] T e61e356d91c645d3ac3627e53a917aa9 P c917ff8588a5472a9c0e61c17cad7bec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:56:37.322094 30764 tablet_replica.cc:439] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8d93c9972cb4cb3b19900fa539c854a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } } }
I20260927 13:56:37.322389 30764 tablet_replica.cc:442] T e61e356d91c645d3ac3627e53a917aa9 P ebffa94065a949fdbad26729946ae3f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:56:37.332721 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.324351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.22.26.193:46017)
0927 13:56:37.324505 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:37.324511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:37.324528 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:56:37.324620 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:37.324625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:37.324748 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:37.324927 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:56:37.325788 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.326928 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.329860 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.330503 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.330626 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.331253 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:56:37.332006 (+ 753us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:56:37.332163 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:37.332328 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 13:56:37.332413 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:56:37.355993 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.348922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:41417 (local address 127.22.26.193:52109)
0927 13:56:37.349603 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:37.349631 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:37.349777 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:37.350147 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:37.350152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:37.350165 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:56:37.350470 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.350482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.352066 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.352072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.353744 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.353756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.354853 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.354857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.354976 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.355026 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:56:37.355532 (+ 506us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:56:37.355547 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 13:56:37.355610 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 13:56:37.355716 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":352,"threads_started":1}
I20260927 13:56:37.356057 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.348917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:52109 (local address 127.22.26.195:41417)
0927 13:56:37.349111 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:37.349117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:37.349730 (+ 613us) server_negotiation.cc:407] Connection header received
0927 13:56:37.349910 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:37.349916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:37.350009 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:37.350117 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:56:37.350663 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.351896 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.353884 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.354685 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.354810 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.355282 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:56:37.355426 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:56:37.355523 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:37.355663 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 13:56:37.355751 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:56:37.366415 30685 raft_consensus.cc:1261] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Refusing update from remote peer ebffa94065a949fdbad26729946ae3f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:37.366498 30542 raft_consensus.cc:1261] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Refusing update from remote peer ebffa94065a949fdbad26729946ae3f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:37.367116 30787 consensus_queue.cc:1035] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:37.367563 30764 consensus_queue.cc:1035] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:37.403713 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.397108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:53255 (local address 127.22.26.193:46017)
0927 13:56:37.397273 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:37.397281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:37.397866 (+ 585us) server_negotiation.cc:407] Connection header received
0927 13:56:37.398096 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:37.398101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:37.398199 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:37.398324 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:56:37.398986 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.399918 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.401540 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.402228 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.402357 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.402890 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:56:37.403052 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:56:37.403182 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:37.403309 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 13:56:37.403387 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 13:56:37.403925 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.397008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:46017 (local address 127.22.26.195:53255)
0927 13:56:37.397774 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:37.397796 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:37.397944 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:37.398329 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:37.398334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:37.398348 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:56:37.398806 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.398823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.400057 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.400060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.401397 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:37.401413 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.402425 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.402434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:37.402621 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.402686 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:56:37.403182 (+ 496us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:56:37.403196 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 13:56:37.403288 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 13:56:37.403421 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":431,"threads_started":1}
I20260927 13:56:37.454475 30245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:37.454799 30245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:37.458642 30245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100: Sending DeleteTablet for 3 replicas of tablet c5e4fdff7f9b406580e557b365b9dbc6
I20260927 13:56:37.460161 30522 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e4fdff7f9b406580e557b365b9dbc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:37 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:38354
I20260927 13:56:37.460549 30383 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e4fdff7f9b406580e557b365b9dbc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:37 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:53106
I20260927 13:56:37.460593 30660 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e4fdff7f9b406580e557b365b9dbc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:37 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:49286
I20260927 13:56:37.466143 30818 tablet_replica.cc:330] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec: stopping tablet replica
I20260927 13:56:37.466461 30818 raft_consensus.cc:2229] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:37.466740 30818 raft_consensus.cc:2258] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:37.472014 30818 ts_tablet_manager.cc:1905] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:37.480069 30818 ts_tablet_manager.cc:1918] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:56:37.480253 30818 log.cc:1196] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/c5e4fdff7f9b406580e557b365b9dbc6
I20260927 13:56:37.480742 30818 ts_tablet_manager.cc:1939] T c5e4fdff7f9b406580e557b365b9dbc6 P c917ff8588a5472a9c0e61c17cad7bec: Deleting consensus metadata
I20260927 13:56:37.482066 30233 catalog_manager.cc:4841] TS c917ff8588a5472a9c0e61c17cad7bec (127.22.26.194:44073): tablet c5e4fdff7f9b406580e557b365b9dbc6 (table test-table [id=c60eaece608d4044a671bc1aa054825f]) successfully deleted
I20260927 13:56:37.488446 30819 tablet_replica.cc:330] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5: stopping tablet replica
I20260927 13:56:37.488651 30819 raft_consensus.cc:2229] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:56:37.489218 30819 raft_consensus.cc:2258] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:37.489845 30820 tablet_replica.cc:330] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a: stopping tablet replica
I20260927 13:56:37.490048 30820 raft_consensus.cc:2229] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:37.490327 30820 raft_consensus.cc:2258] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:37.495669 30819 ts_tablet_manager.cc:1905] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:37.498569 30820 ts_tablet_manager.cc:1905] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:37.513448 30820 ts_tablet_manager.cc:1918] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:56:37.513610 30820 log.cc:1196] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/c5e4fdff7f9b406580e557b365b9dbc6
I20260927 13:56:37.518451 30820 ts_tablet_manager.cc:1939] T c5e4fdff7f9b406580e557b365b9dbc6 P b8d93c9972cb4cb3b19900fa539c854a: Deleting consensus metadata
I20260927 13:56:37.519892 30819 ts_tablet_manager.cc:1918] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:56:37.520028 30819 log.cc:1196] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/c5e4fdff7f9b406580e557b365b9dbc6
I20260927 13:56:37.520474 30819 ts_tablet_manager.cc:1939] T c5e4fdff7f9b406580e557b365b9dbc6 P ebffa94065a949fdbad26729946ae3f5: Deleting consensus metadata
I20260927 13:56:37.531025 30233 catalog_manager.cc:4841] TS b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195:41417): tablet c5e4fdff7f9b406580e557b365b9dbc6 (table test-table [id=c60eaece608d4044a671bc1aa054825f]) successfully deleted
I20260927 13:56:37.537457 30232 catalog_manager.cc:4841] TS ebffa94065a949fdbad26729946ae3f5 (127.22.26.193:46017): tablet c5e4fdff7f9b406580e557b365b9dbc6 (table test-table [id=c60eaece608d4044a671bc1aa054825f]) successfully deleted
Sep 27 13:56:37 dist-test-slave-gkks krb5kdc[30201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:56:37 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:37 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:56:37.612946 30795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:37.591737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.22.26.254:43645)
0927 13:56:37.591949 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:37.591957 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:37.599456 (+ 7499us) server_negotiation.cc:407] Connection header received
0927 13:56:37.599775 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:37.599787 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:37.599876 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:37.599953 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:37.601892 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.603017 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.604927 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.605587 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.605697 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.612658 (+ 6961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45548: BlockingRecv error: recv got EOF from 127.0.0.1:45548 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 13:56:37 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM
I20260927 13:56:37.671591 30795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.644886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.22.26.254:43645)
0927 13:56:37.645098 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:37.645105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:37.652313 (+ 7208us) server_negotiation.cc:407] Connection header received
0927 13:56:37.652916 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:37.652922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:37.653015 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:37.653087 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:37.654951 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.656123 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.658118 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.658813 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.658927 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.664447 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:37.664457 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:37.664464 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:37.664512 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:37.668121 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:37.669040 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:37.669046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:37.669050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:37.669215 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:37.669704 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:37.669709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:37.669713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:37.670084 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:37.670319 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:37.670917 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 13:56:37.671212 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:56:37.676465 30245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:56:37.677227 30245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:56:37.685163 30383 tablet_service.cc:1467] Processing CreateTablet for tablet 4c339c8f33ea4ead9b593d1dff4be71c (DEFAULT_TABLE table=test-table [id=dd856b9ebf564989bbbf6c55cfed98bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:37.685652 30383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c339c8f33ea4ead9b593d1dff4be71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:37.692049 30522 tablet_service.cc:1467] Processing CreateTablet for tablet 4c339c8f33ea4ead9b593d1dff4be71c (DEFAULT_TABLE table=test-table [id=dd856b9ebf564989bbbf6c55cfed98bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:37.692502 30522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c339c8f33ea4ead9b593d1dff4be71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:37.695847 30660 tablet_service.cc:1467] Processing CreateTablet for tablet 4c339c8f33ea4ead9b593d1dff4be71c (DEFAULT_TABLE table=test-table [id=dd856b9ebf564989bbbf6c55cfed98bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:37.696343 30660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c339c8f33ea4ead9b593d1dff4be71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:37.703363 30758 tablet_bootstrap.cc:492] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5: Bootstrap starting.
I20260927 13:56:37.703657 30757 tablet_bootstrap.cc:492] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec: Bootstrap starting.
I20260927 13:56:37.705140 30757 tablet_bootstrap.cc:654] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:37.705444 30758 tablet_bootstrap.cc:654] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:37.707333 30757 tablet_bootstrap.cc:492] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec: No bootstrap required, opened a new log
I20260927 13:56:37.707489 30757 ts_tablet_manager.cc:1397] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260927 13:56:37.708779 30756 tablet_bootstrap.cc:492] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a: Bootstrap starting.
I20260927 13:56:37.709481 30757 raft_consensus.cc:348] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.709766 30757 raft_consensus.cc:374] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:37.709864 30757 raft_consensus.cc:729] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c917ff8588a5472a9c0e61c17cad7bec, State: Initialized, Role: FOLLOWER
I20260927 13:56:37.710672 30756 tablet_bootstrap.cc:654] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:37.710572 30757 consensus_queue.cc:260] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.711814 30757 ts_tablet_manager.cc:1428] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 13:56:37.720383 30758 tablet_bootstrap.cc:492] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5: No bootstrap required, opened a new log
I20260927 13:56:37.720414 30756 tablet_bootstrap.cc:492] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a: No bootstrap required, opened a new log
I20260927 13:56:37.720556 30756 ts_tablet_manager.cc:1397] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20260927 13:56:37.720566 30758 ts_tablet_manager.cc:1397] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.004s
I20260927 13:56:37.721385 30758 raft_consensus.cc:348] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.721376 30756 raft_consensus.cc:348] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.721657 30756 raft_consensus.cc:374] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:37.721673 30758 raft_consensus.cc:374] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:37.721764 30756 raft_consensus.cc:729] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d93c9972cb4cb3b19900fa539c854a, State: Initialized, Role: FOLLOWER
I20260927 13:56:37.721772 30758 raft_consensus.cc:729] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebffa94065a949fdbad26729946ae3f5, State: Initialized, Role: FOLLOWER
I20260927 13:56:37.722039 30758 consensus_queue.cc:260] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.722029 30756 consensus_queue.cc:260] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.722566 30756 ts_tablet_manager.cc:1428] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 13:56:37.723387 30758 ts_tablet_manager.cc:1428] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 13:56:37.747378 30801 raft_consensus.cc:482] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:56:37.747591 30801 raft_consensus.cc:504] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.748504 30801 leader_election.cc:288] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebffa94065a949fdbad26729946ae3f5 (127.22.26.193:46017), c917ff8588a5472a9c0e61c17cad7bec (127.22.26.194:44073)
I20260927 13:56:37.749318 30541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c339c8f33ea4ead9b593d1dff4be71c" candidate_uuid: "b8d93c9972cb4cb3b19900fa539c854a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c917ff8588a5472a9c0e61c17cad7bec" is_pre_election: true
I20260927 13:56:37.749598 30541 raft_consensus.cc:2454] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8d93c9972cb4cb3b19900fa539c854a in term 0.
I20260927 13:56:37.749780 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c339c8f33ea4ead9b593d1dff4be71c" candidate_uuid: "b8d93c9972cb4cb3b19900fa539c854a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebffa94065a949fdbad26729946ae3f5" is_pre_election: true
I20260927 13:56:37.750051 30407 raft_consensus.cc:2454] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8d93c9972cb4cb3b19900fa539c854a in term 0.
I20260927 13:56:37.750417 30615 leader_election.cc:302] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d93c9972cb4cb3b19900fa539c854a, c917ff8588a5472a9c0e61c17cad7bec; no voters:
I20260927 13:56:37.751420 30801 raft_consensus.cc:2781] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:56:37.751534 30801 raft_consensus.cc:482] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:56:37.751652 30801 raft_consensus.cc:3037] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:37.754774 30801 raft_consensus.cc:504] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.757143 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c339c8f33ea4ead9b593d1dff4be71c" candidate_uuid: "b8d93c9972cb4cb3b19900fa539c854a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebffa94065a949fdbad26729946ae3f5"
I20260927 13:56:37.757395 30407 raft_consensus.cc:3037] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:37.760802 30407 raft_consensus.cc:2454] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d93c9972cb4cb3b19900fa539c854a in term 1.
I20260927 13:56:37.761250 30614 leader_election.cc:302] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d93c9972cb4cb3b19900fa539c854a, ebffa94065a949fdbad26729946ae3f5; no voters:
I20260927 13:56:37.761572 30811 raft_consensus.cc:2781] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:37.761740 30811 raft_consensus.cc:686] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [term 1 LEADER]: Becoming Leader. State: Replica: b8d93c9972cb4cb3b19900fa539c854a, State: Running, Role: LEADER
I20260927 13:56:37.762063 30811 consensus_queue.cc:237] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.762220 30541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c339c8f33ea4ead9b593d1dff4be71c" candidate_uuid: "b8d93c9972cb4cb3b19900fa539c854a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c917ff8588a5472a9c0e61c17cad7bec"
I20260927 13:56:37.762446 30541 raft_consensus.cc:3037] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:37.766705 30541 raft_consensus.cc:2454] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d93c9972cb4cb3b19900fa539c854a in term 1.
I20260927 13:56:37.767261 30801 leader_election.cc:288] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [CANDIDATE]: Term 1 election: Requested vote from peers ebffa94065a949fdbad26729946ae3f5 (127.22.26.193:46017), c917ff8588a5472a9c0e61c17cad7bec (127.22.26.194:44073)
I20260927 13:56:37.776719 30245 catalog_manager.cc:5495] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a reported cstate change: term changed from 0 to 1, leader changed from <none> to b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195). New cstate: current_term: 1 leader_uuid: "b8d93c9972cb4cb3b19900fa539c854a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } health_report { overall_health: HEALTHY } } }
I20260927 13:56:37.812110 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:37.804740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.22.26.195:41417)
0927 13:56:37.805401 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:37.805407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:37.805426 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:56:37.805504 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:37.805509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:37.805594 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:37.805681 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:56:37.806261 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.807420 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.809519 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:37.810176 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:37.810320 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:37.810892 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:56:37.811058 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:56:37.811178 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:37.811726 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 13:56:37.811813 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527}
I20260927 13:56:37.817282 30541 raft_consensus.cc:1261] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Refusing update from remote peer b8d93c9972cb4cb3b19900fa539c854a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:37.818028 30407 raft_consensus.cc:1261] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Refusing update from remote peer b8d93c9972cb4cb3b19900fa539c854a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:37.818776 30801 consensus_queue.cc:1035] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:37.819799 30801 consensus_queue.cc:1035] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:37.847241 30245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:37.847509 30245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:37.851413 30245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100: Sending DeleteTablet for 3 replicas of tablet 4c339c8f33ea4ead9b593d1dff4be71c
I20260927 13:56:37.852241 30383 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c339c8f33ea4ead9b593d1dff4be71c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:37 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:53106
I20260927 13:56:37.852552 30819 tablet_replica.cc:330] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5: stopping tablet replica
I20260927 13:56:37.852793 30819 raft_consensus.cc:2229] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:37.853055 30819 raft_consensus.cc:2258] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:37.854446 30819 ts_tablet_manager.cc:1905] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:37.864100 30819 ts_tablet_manager.cc:1918] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:37.864274 30819 log.cc:1196] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/4c339c8f33ea4ead9b593d1dff4be71c
I20260927 13:56:37.864813 30819 ts_tablet_manager.cc:1939] T 4c339c8f33ea4ead9b593d1dff4be71c P ebffa94065a949fdbad26729946ae3f5: Deleting consensus metadata
I20260927 13:56:37.866030 30232 catalog_manager.cc:4841] TS ebffa94065a949fdbad26729946ae3f5 (127.22.26.193:46017): tablet 4c339c8f33ea4ead9b593d1dff4be71c (table test-table [id=dd856b9ebf564989bbbf6c55cfed98bb]) successfully deleted
I20260927 13:56:37.867046 30522 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c339c8f33ea4ead9b593d1dff4be71c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:37 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:38354
I20260927 13:56:37.867419 30818 tablet_replica.cc:330] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec: stopping tablet replica
I20260927 13:56:37.867677 30818 raft_consensus.cc:2229] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:37.867969 30818 raft_consensus.cc:2258] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:37.867980 30660 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c339c8f33ea4ead9b593d1dff4be71c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:37 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:49286
I20260927 13:56:37.868284 30820 tablet_replica.cc:330] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a: stopping tablet replica
I20260927 13:56:37.868530 30820 raft_consensus.cc:2229] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:56:37.869082 30820 raft_consensus.cc:2258] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:37.869266 30818 ts_tablet_manager.cc:1905] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:37.871170 30246 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:56:37.871927 30246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:56:37.875178 30820 ts_tablet_manager.cc:1905] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:37.881018 30383 tablet_service.cc:1467] Processing CreateTablet for tablet b27220196fc04c2c967c5824fefb1509 (DEFAULT_TABLE table=test-table [id=078f7ea3eb1749319debf7e9ac6c43fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:37.881502 30383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27220196fc04c2c967c5824fefb1509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:37.884346 30522 tablet_service.cc:1467] Processing CreateTablet for tablet b27220196fc04c2c967c5824fefb1509 (DEFAULT_TABLE table=test-table [id=078f7ea3eb1749319debf7e9ac6c43fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:37.884631 30820 ts_tablet_manager.cc:1918] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:37.884784 30522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27220196fc04c2c967c5824fefb1509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:37.884819 30820 log.cc:1196] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/4c339c8f33ea4ead9b593d1dff4be71c
I20260927 13:56:37.885510 30820 ts_tablet_manager.cc:1939] T 4c339c8f33ea4ead9b593d1dff4be71c P b8d93c9972cb4cb3b19900fa539c854a: Deleting consensus metadata
I20260927 13:56:37.887619 30660 tablet_service.cc:1467] Processing CreateTablet for tablet b27220196fc04c2c967c5824fefb1509 (DEFAULT_TABLE table=test-table [id=078f7ea3eb1749319debf7e9ac6c43fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:56:37.888022 30660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27220196fc04c2c967c5824fefb1509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:56:37.891520 30233 catalog_manager.cc:4841] TS b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195:41417): tablet 4c339c8f33ea4ead9b593d1dff4be71c (table test-table [id=dd856b9ebf564989bbbf6c55cfed98bb]) successfully deleted
I20260927 13:56:37.913375 30818 ts_tablet_manager.cc:1918] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:56:37.914633 30757 tablet_bootstrap.cc:492] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec: Bootstrap starting.
I20260927 13:56:37.916391 30757 tablet_bootstrap.cc:654] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:37.931944 30756 tablet_bootstrap.cc:492] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a: Bootstrap starting.
I20260927 13:56:37.932495 30818 log.cc:1196] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/4c339c8f33ea4ead9b593d1dff4be71c
I20260927 13:56:37.932770 30758 tablet_bootstrap.cc:492] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5: Bootstrap starting.
I20260927 13:56:37.933110 30818 ts_tablet_manager.cc:1939] T 4c339c8f33ea4ead9b593d1dff4be71c P c917ff8588a5472a9c0e61c17cad7bec: Deleting consensus metadata
I20260927 13:56:37.934640 30758 tablet_bootstrap.cc:654] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:37.937626 30756 tablet_bootstrap.cc:654] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:37.940644 30233 catalog_manager.cc:4841] TS c917ff8588a5472a9c0e61c17cad7bec (127.22.26.194:44073): tablet 4c339c8f33ea4ead9b593d1dff4be71c (table test-table [id=dd856b9ebf564989bbbf6c55cfed98bb]) successfully deleted
I20260927 13:56:37.944474 30757 tablet_bootstrap.cc:492] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec: No bootstrap required, opened a new log
I20260927 13:56:37.944669 30757 ts_tablet_manager.cc:1397] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20260927 13:56:37.945531 30757 raft_consensus.cc:348] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.945820 30757 raft_consensus.cc:374] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:37.945904 30757 raft_consensus.cc:729] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c917ff8588a5472a9c0e61c17cad7bec, State: Initialized, Role: FOLLOWER
I20260927 13:56:37.946214 30757 consensus_queue.cc:260] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.949903 30757 ts_tablet_manager.cc:1428] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260927 13:56:37.971441 30756 tablet_bootstrap.cc:492] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a: No bootstrap required, opened a new log
I20260927 13:56:37.971644 30756 ts_tablet_manager.cc:1397] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.005s
I20260927 13:56:37.972545 30756 raft_consensus.cc:348] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.972828 30756 raft_consensus.cc:374] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:37.972918 30756 raft_consensus.cc:729] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d93c9972cb4cb3b19900fa539c854a, State: Initialized, Role: FOLLOWER
I20260927 13:56:37.973238 30756 consensus_queue.cc:260] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.976682 30756 ts_tablet_manager.cc:1428] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 13:56:37.978328 30758 tablet_bootstrap.cc:492] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5: No bootstrap required, opened a new log
I20260927 13:56:37.978524 30758 ts_tablet_manager.cc:1397] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.010s
I20260927 13:56:37.979444 30758 raft_consensus.cc:348] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.979743 30758 raft_consensus.cc:374] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:37.979830 30758 raft_consensus.cc:729] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebffa94065a949fdbad26729946ae3f5, State: Initialized, Role: FOLLOWER
I20260927 13:56:37.980118 30758 consensus_queue.cc:260] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:37.984206 30758 ts_tablet_manager.cc:1428] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260927 13:56:38.039240 30767 raft_consensus.cc:482] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:56:38.039479 30767 raft_consensus.cc:504] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:38.040177 30767 leader_election.cc:288] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebffa94065a949fdbad26729946ae3f5 (127.22.26.193:46017), b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195:41417)
I20260927 13:56:38.040989 30685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27220196fc04c2c967c5824fefb1509" candidate_uuid: "c917ff8588a5472a9c0e61c17cad7bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d93c9972cb4cb3b19900fa539c854a" is_pre_election: true
I20260927 13:56:38.041265 30685 raft_consensus.cc:2454] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c917ff8588a5472a9c0e61c17cad7bec in term 0.
I20260927 13:56:38.042115 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27220196fc04c2c967c5824fefb1509" candidate_uuid: "c917ff8588a5472a9c0e61c17cad7bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebffa94065a949fdbad26729946ae3f5" is_pre_election: true
I20260927 13:56:38.042400 30407 raft_consensus.cc:2454] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c917ff8588a5472a9c0e61c17cad7bec in term 0.
I20260927 13:56:38.044083 30476 leader_election.cc:302] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d93c9972cb4cb3b19900fa539c854a, c917ff8588a5472a9c0e61c17cad7bec; no voters:
I20260927 13:56:38.044960 30767 raft_consensus.cc:2781] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:56:38.045064 30767 raft_consensus.cc:482] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:56:38.045176 30767 raft_consensus.cc:3037] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:38.048171 30767 raft_consensus.cc:504] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:38.049280 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27220196fc04c2c967c5824fefb1509" candidate_uuid: "c917ff8588a5472a9c0e61c17cad7bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebffa94065a949fdbad26729946ae3f5"
I20260927 13:56:38.049513 30407 raft_consensus.cc:3037] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:38.051379 30767 leader_election.cc:288] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [CANDIDATE]: Term 1 election: Requested vote from peers ebffa94065a949fdbad26729946ae3f5 (127.22.26.193:46017), b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195:41417)
I20260927 13:56:38.052028 30685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27220196fc04c2c967c5824fefb1509" candidate_uuid: "c917ff8588a5472a9c0e61c17cad7bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d93c9972cb4cb3b19900fa539c854a"
I20260927 13:56:38.052284 30685 raft_consensus.cc:3037] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:38.053491 30407 raft_consensus.cc:2454] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c917ff8588a5472a9c0e61c17cad7bec in term 1.
I20260927 13:56:38.053903 30475 leader_election.cc:302] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c917ff8588a5472a9c0e61c17cad7bec, ebffa94065a949fdbad26729946ae3f5; no voters:
I20260927 13:56:38.054234 30767 raft_consensus.cc:2781] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:38.055887 30685 raft_consensus.cc:2454] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c917ff8588a5472a9c0e61c17cad7bec in term 1.
I20260927 13:56:38.059734 30767 raft_consensus.cc:686] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [term 1 LEADER]: Becoming Leader. State: Replica: c917ff8588a5472a9c0e61c17cad7bec, State: Running, Role: LEADER
I20260927 13:56:38.060353 30767 consensus_queue.cc:237] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } }
I20260927 13:56:38.070336 30246 catalog_manager.cc:5495] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec reported cstate change: term changed from 0 to 1, leader changed from <none> to c917ff8588a5472a9c0e61c17cad7bec (127.22.26.194). New cstate: current_term: 1 leader_uuid: "c917ff8588a5472a9c0e61c17cad7bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c917ff8588a5472a9c0e61c17cad7bec" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 44073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 } health_report { overall_health: UNKNOWN } } }
I20260927 13:56:38.154819 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:38.133442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.22.26.194:44073)
0927 13:56:38.146840 (+ 13398us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:38.146848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:38.146875 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:56:38.146963 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:38.146969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:38.147061 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:38.147869 (+ 808us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:56:38.148060 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.149248 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.151628 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.152358 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.153125 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:38.153258 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:56:38.153413 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:56:38.154214 (+ 801us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:38.154278 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 13:56:38.154369 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13239,"thread_start_us":13795,"threads_started":1}
I20260927 13:56:38.202625 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:38.189591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:50917 (local address 127.22.26.195:41417)
0927 13:56:38.189807 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:38.189814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:38.195757 (+ 5943us) server_negotiation.cc:407] Connection header received
0927 13:56:38.195923 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:38.195928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:38.196014 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:38.196113 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:56:38.196792 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.197918 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.200297 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.200981 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.201134 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:38.201717 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:56:38.201874 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:56:38.201972 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:38.202178 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 13:56:38.202270 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:56:38.202997 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:38.189174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:41417 (local address 127.22.26.194:50917)
0927 13:56:38.195612 (+ 6438us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:38.195631 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:38.195786 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:38.196190 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:38.196195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:38.196210 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:56:38.196571 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:38.196599 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.198070 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.198076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:38.200113 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:38.200134 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.201209 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.201218 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:38.201408 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:38.201493 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:56:38.202014 (+ 521us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:56:38.202029 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 13:56:38.202134 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 13:56:38.202295 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6235,"thread_start_us":6086,"threads_started":1}
I20260927 13:56:38.214352 30685 raft_consensus.cc:1261] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Refusing update from remote peer c917ff8588a5472a9c0e61c17cad7bec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:38.214500 30407 raft_consensus.cc:1261] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Refusing update from remote peer c917ff8588a5472a9c0e61c17cad7bec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:56:38.215059 30843 consensus_queue.cc:1035] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8d93c9972cb4cb3b19900fa539c854a" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 41417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:38.217113 30843 consensus_queue.cc:1035] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebffa94065a949fdbad26729946ae3f5" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:56:38.256072 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:38.248878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:32897 (local address 127.22.26.194:44073)
0927 13:56:38.249078 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:38.249086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:38.249636 (+ 550us) server_negotiation.cc:407] Connection header received
0927 13:56:38.249865 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:38.249871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:38.249953 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:38.250066 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:56:38.250941 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.252171 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.253967 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.254416 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.254586 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:38.255171 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:56:38.255316 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:56:38.255386 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:38.255654 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 13:56:38.255751 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:56:38.257019 30852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:38.248591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:44073 (local address 127.22.26.195:32897)
0927 13:56:38.249503 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:38.249525 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:38.249705 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:38.250305 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:38.250310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:38.250324 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:56:38.250723 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:38.250739 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.252403 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.252408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:38.253789 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:38.253803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.254653 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.254660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:38.254870 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:38.254933 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:56:38.255482 (+ 549us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:56:38.255492 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 13:56:38.256348 (+ 856us) client_negotiation.cc:240] Negotiation successful
0927 13:56:38.256519 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":555,"threads_started":1}
I20260927 13:56:38.300239 30245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:38.300529 30245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:56:38.303725 30245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63065962bd344208bfbd4a38a19a6100: Sending DeleteTablet for 3 replicas of tablet b27220196fc04c2c967c5824fefb1509
I20260927 13:56:38.304579 30383 tablet_service.cc:1514] Processing DeleteTablet for tablet b27220196fc04c2c967c5824fefb1509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:38 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:53106
I20260927 13:56:38.304961 30522 tablet_service.cc:1514] Processing DeleteTablet for tablet b27220196fc04c2c967c5824fefb1509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:38 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:38354
I20260927 13:56:38.305130 30819 tablet_replica.cc:330] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5: stopping tablet replica
I20260927 13:56:38.305250 30818 tablet_replica.cc:330] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec: stopping tablet replica
I20260927 13:56:38.305331 30819 raft_consensus.cc:2229] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:38.305416 30818 raft_consensus.cc:2229] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:56:38.305536 30819 raft_consensus.cc:2258] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:38.305752 30818 raft_consensus.cc:2258] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:38.306025 30660 tablet_service.cc:1514] Processing DeleteTablet for tablet b27220196fc04c2c967c5824fefb1509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:56:38 UTC) from {username='oryx', principal='oryx/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:49286
I20260927 13:56:38.306879 30819 ts_tablet_manager.cc:1905] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:38.307365 30818 ts_tablet_manager.cc:1905] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:38.312577 30818 ts_tablet_manager.cc:1918] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:56:38.312778 30818 log.cc:1196] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/b27220196fc04c2c967c5824fefb1509
I20260927 13:56:38.313181 30818 ts_tablet_manager.cc:1939] T b27220196fc04c2c967c5824fefb1509 P c917ff8588a5472a9c0e61c17cad7bec: Deleting consensus metadata
I20260927 13:56:38.313889 30819 ts_tablet_manager.cc:1918] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:56:38.314018 30819 log.cc:1196] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/b27220196fc04c2c967c5824fefb1509
I20260927 13:56:38.314117 30820 tablet_replica.cc:330] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a: stopping tablet replica
I20260927 13:56:38.314358 30820 raft_consensus.cc:2229] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:56:38.314468 30819 ts_tablet_manager.cc:1939] T b27220196fc04c2c967c5824fefb1509 P ebffa94065a949fdbad26729946ae3f5: Deleting consensus metadata
I20260927 13:56:38.314881 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 30317
I20260927 13:56:38.315133 30233 catalog_manager.cc:4841] TS c917ff8588a5472a9c0e61c17cad7bec (127.22.26.194:44073): tablet b27220196fc04c2c967c5824fefb1509 (table test-table [id=078f7ea3eb1749319debf7e9ac6c43fa]) successfully deleted
I20260927 13:56:38.315492 30820 raft_consensus.cc:2258] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:56:38.316109 30232 catalog_manager.cc:4841] TS ebffa94065a949fdbad26729946ae3f5 (127.22.26.193:46017): tablet b27220196fc04c2c967c5824fefb1509 (table test-table [id=078f7ea3eb1749319debf7e9ac6c43fa]) successfully deleted
I20260927 13:56:38.317246 30820 ts_tablet_manager.cc:1905] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:56:38.324193 30820 ts_tablet_manager.cc:1918] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:56:38.324325 30820 log.cc:1196] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/b27220196fc04c2c967c5824fefb1509
I20260927 13:56:38.324769 30820 ts_tablet_manager.cc:1939] T b27220196fc04c2c967c5824fefb1509 P b8d93c9972cb4cb3b19900fa539c854a: Deleting consensus metadata
I20260927 13:56:38.326088 30233 catalog_manager.cc:4841] TS b8d93c9972cb4cb3b19900fa539c854a (127.22.26.195:41417): tablet b27220196fc04c2c967c5824fefb1509 (table test-table [id=078f7ea3eb1749319debf7e9ac6c43fa]) successfully deleted
Sep 27 13:56:38 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.26.193@KRBTEST.COM
I20260927 13:56:38.343475 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:38.318198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53124 (local address 127.22.26.193:46017)
0927 13:56:38.326312 (+ 8114us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:38.326318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:38.326891 (+ 573us) server_negotiation.cc:407] Connection header received
0927 13:56:38.326953 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:38.326958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:38.327032 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:38.327167 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:38.328970 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.330075 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.331954 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.332495 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.332599 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:38.337591 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:38.337603 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:38.337619 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:38.337653 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:38.340616 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:38.341343 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:38.341351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:38.341353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:38.341480 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:38.341889 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:38.341900 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:38.341902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:38.342254 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:38.342584 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:38.342862 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 13:56:38.343124 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7972,"thread_start_us":8135,"threads_started":1}
I20260927 13:56:38.344218 30448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:38.471347 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 30317
I20260927 13:56:38.501500 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 30457
Sep 27 13:56:38 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.26.194@KRBTEST.COM
I20260927 13:56:38.517055 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:38.502405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.22.26.194:44073)
0927 13:56:38.502574 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:38.502578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:38.502777 (+ 199us) server_negotiation.cc:407] Connection header received
0927 13:56:38.503009 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:38.503015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:38.503147 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:38.503277 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:38.504510 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.505211 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.506492 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.506919 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.507061 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:38.511338 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:38.511351 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:38.511366 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:38.511400 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:38.514448 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:38.515314 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:38.515321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:38.515324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:38.515446 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:38.515792 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:38.515796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:38.515798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:38.516118 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:38.516353 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:38.516644 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 13:56:38.516806 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 13:56:38.517699 30587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:38.644492 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 30457
I20260927 13:56:38.666904 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 30596
Sep 27 13:56:38 dist-test-slave-gkks krb5kdc[30201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.26.195@KRBTEST.COM
I20260927 13:56:38.681427 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:38.667764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.22.26.195:41417)
0927 13:56:38.667923 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:38.667928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:38.667944 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:56:38.668014 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:38.668018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:38.668083 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:38.668168 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:38.669383 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.670078 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.671446 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:38.671857 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:38.671947 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:38.676091 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:38.676105 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:38.676120 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:38.676152 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:38.678983 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:38.679614 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:38.679621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:38.679624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:38.679750 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:38.680159 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:38.680162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:38.680164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:38.680506 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:38.680741 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:38.681023 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 13:56:38.681179 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:56:38.682102 30726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:38.830331 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 30596
I20260927 13:56:38.855917 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 30214
I20260927 13:56:38.857219 30286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:38.976325 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 30214
2026-09-27T13:56:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6959 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:56:39 dist-test-slave-gkks krb5kdc[30865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:56:39 dist-test-slave-gkks krb5kdc[30865](info): set up 2 sockets
Sep 27 13:56:39 dist-test-slave-gkks krb5kdc[30865](info): commencing operation
krb5kdc: starting...
W20260927 13:56:40.076220 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:56:40 dist-test-slave-gkks krb5kdc[30865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:56:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:56:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:56:40.286829 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:36109
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:36109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:40.602408 30878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:40.602732 30878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:40.602917 30878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:40.609884 30878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:40.610040 30878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:40.610158 30878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:40.610247 30878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:40.610318 30878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:56:40.619663 30878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:36109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:36109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:40.620918 30878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:40.622193 30878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:40.633643 30884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:40.633705 30883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:40.633948 30886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:40.634338 30878 server_base.cc:1034] running on GCE node
I20260927 13:56:40.635023 30878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:40.636448 30878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:40.637688 30878 hybrid_clock.cc:648] HybridClock initialized: now 1790517400637647 us; error 71 us; skew 500 ppm
I20260927 13:56:40.638011 30878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T13:56:40Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1732 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:56:40 dist-test-slave-gkks krb5kdc[30892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:56:40 dist-test-slave-gkks krb5kdc[30892](info): set up 2 sockets
Sep 27 13:56:40 dist-test-slave-gkks krb5kdc[30892](info): commencing operation
krb5kdc: starting...
W20260927 13:56:41.820174 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:56:41 dist-test-slave-gkks krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:56:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:56:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:56:42.042537 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:44809
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:44809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:42.369230 30905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:42.369515 30905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:42.369678 30905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:42.376823 30905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:42.376915 30905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:42.376998 30905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:42.377108 30905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:42.377175 30905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:56:42.386936 30905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:44809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:44809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:42.388281 30905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:42.389628 30905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:42.401784 30913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:42.401714 30911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:42.402159 30910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:42.402938 30905 server_base.cc:1034] running on GCE node
I20260927 13:56:42.403743 30905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:42.405216 30905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:42.406455 30905 hybrid_clock.cc:648] HybridClock initialized: now 1790517402406410 us; error 39 us; skew 500 ppm
I20260927 13:56:42.406803 30905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:42 dist-test-slave-gkks krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517402, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:42.409924 30905 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:56:42.411340 30905 webserver.cc:460] Webserver started at http://127.22.26.254:34329/ using document root <none> and password file <none>
I20260927 13:56:42.412010 30905 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:42.412160 30905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:42.412492 30905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:42.415452 30905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "6100d33e90a04a719f6f71cbbcbefbad"
format_stamp: "Formatted at 2026-09-27 13:56:42 on dist-test-slave-gkks"
server_key: "3af7f3d3029ac4cd37535e3e98dad2ff"
server_key_iv: "066015f970ede8213848c761e47b0956"
server_key_version: "encryptionkey@0"
I20260927 13:56:42.416195 30905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "6100d33e90a04a719f6f71cbbcbefbad"
format_stamp: "Formatted at 2026-09-27 13:56:42 on dist-test-slave-gkks"
server_key: "3af7f3d3029ac4cd37535e3e98dad2ff"
server_key_iv: "066015f970ede8213848c761e47b0956"
server_key_version: "encryptionkey@0"
I20260927 13:56:42.421458 30905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:56:42.424829 30920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:42.426070 30905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:56:42.426272 30905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "6100d33e90a04a719f6f71cbbcbefbad"
format_stamp: "Formatted at 2026-09-27 13:56:42 on dist-test-slave-gkks"
server_key: "3af7f3d3029ac4cd37535e3e98dad2ff"
server_key_iv: "066015f970ede8213848c761e47b0956"
server_key_version: "encryptionkey@0"
I20260927 13:56:42.426474 30905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:42.448666 30905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:42.453022 30905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:42.453368 30905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:42.471349 30905 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:44809
I20260927 13:56:42.471356 30971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:44809 every 8 connection(s)
I20260927 13:56:42.472692 30905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:56:42.474048 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 30905
I20260927 13:56:42.474277 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:56:42.475075 22635 external_mini_cluster.cc:1428] Setting key 10ddd9f928b0eee71d797414b2f0f8d5
I20260927 13:56:42.477409 30972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:56:42 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:42.490612 30972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad: Bootstrap starting.
I20260927 13:56:42.493745 30972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad: Neither blocks nor log segments found. Creating new log.
I20260927 13:56:42.494781 30972 log.cc:824] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:42.498164 30972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad: No bootstrap required, opened a new log
I20260927 13:56:42.498385 30975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:42.477126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44902 (local address 127.22.26.254:44809)
0927 13:56:42.477921 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:42.477933 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:42.477972 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:56:42.479649 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:42.479676 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:42.480048 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:42.480365 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:42.482193 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:42.483553 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:42.485321 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:42.485988 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:42.486121 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:42.490687 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:42.490695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:42.490708 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:42.490773 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:42.493929 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:42.494649 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:42.494658 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:42.494676 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:42.494835 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:42.495326 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:42.495334 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:42.495339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:42.496135 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:42.496402 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:42.496900 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 13:56:42.497261 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":364,"threads_started":1}
I20260927 13:56:42.506258 30972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6100d33e90a04a719f6f71cbbcbefbad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44809 } }
I20260927 13:56:42.506621 30972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:56:42.506753 30972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6100d33e90a04a719f6f71cbbcbefbad, State: Initialized, Role: FOLLOWER
I20260927 13:56:42.507356 30972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6100d33e90a04a719f6f71cbbcbefbad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44809 } }
I20260927 13:56:42.507594 30972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:56:42.507754 30972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:56:42.507938 30972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:56:42.510169 30972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6100d33e90a04a719f6f71cbbcbefbad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44809 } }
I20260927 13:56:42.510594 30972 leader_election.cc:302] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6100d33e90a04a719f6f71cbbcbefbad; no voters:
I20260927 13:56:42.511155 30972 leader_election.cc:288] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:56:42.511310 30977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:56:42.512012 30977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 1 LEADER]: Becoming Leader. State: Replica: 6100d33e90a04a719f6f71cbbcbefbad, State: Running, Role: LEADER
I20260927 13:56:42.512530 30977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6100d33e90a04a719f6f71cbbcbefbad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44809 } }
I20260927 13:56:42.513160 30972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:42.515486 30978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6100d33e90a04a719f6f71cbbcbefbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6100d33e90a04a719f6f71cbbcbefbad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44809 } } }
I20260927 13:56:42.515834 30978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:42.516323 30979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6100d33e90a04a719f6f71cbbcbefbad. Latest consensus state: current_term: 1 leader_uuid: "6100d33e90a04a719f6f71cbbcbefbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6100d33e90a04a719f6f71cbbcbefbad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44809 } } }
I20260927 13:56:42.516602 30979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:42.522415 30984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:42.527437 30984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:42.536347 30984 catalog_manager.cc:1347] Generated new cluster ID: b996eaa0b2c04845a10328f57df33cea
I20260927 13:56:42.536485 30984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:42.563920 30984 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:56:42.565308 30984 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:42.577888 30984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad: Generated new TSK 0
I20260927 13:56:42.578743 30984 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:56:42.632524 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:44809
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:42.958992 30998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:42.959267 30998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:42.959359 30998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:42.966311 30998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:42.966413 30998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:42.966523 30998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:56:42.976207 30998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:44809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:42.977466 30998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:42.978714 30998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:42.991633 31004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:42.991590 31003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:42.992132 31006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:42.992352 30998 server_base.cc:1034] running on GCE node
I20260927 13:56:42.992960 30998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:42.993872 30998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:42.995121 30998 hybrid_clock.cc:648] HybridClock initialized: now 1790517402995040 us; error 31 us; skew 500 ppm
I20260927 13:56:42.995445 30998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:42 dist-test-slave-gkks krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517402, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:42.998903 30998 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:56:43.000419 30998 webserver.cc:460] Webserver started at http://127.22.26.193:45409/ using document root <none> and password file <none>
I20260927 13:56:43.001036 30998 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:43.001192 30998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:43.001458 30998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:43.004222 30998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "b5b110b410bd45619a27145bdd13755b"
format_stamp: "Formatted at 2026-09-27 13:56:42 on dist-test-slave-gkks"
server_key: "f633889491eed846fb004056dd20c7c5"
server_key_iv: "7e6aa5a2df35960ec4331a6f0bc4dd29"
server_key_version: "encryptionkey@0"
I20260927 13:56:43.004917 30998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "b5b110b410bd45619a27145bdd13755b"
format_stamp: "Formatted at 2026-09-27 13:56:42 on dist-test-slave-gkks"
server_key: "f633889491eed846fb004056dd20c7c5"
server_key_iv: "7e6aa5a2df35960ec4331a6f0bc4dd29"
server_key_version: "encryptionkey@0"
I20260927 13:56:43.009754 30998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 13:56:43.013213 31013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:43.014640 30998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:56:43.014811 30998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "b5b110b410bd45619a27145bdd13755b"
format_stamp: "Formatted at 2026-09-27 13:56:42 on dist-test-slave-gkks"
server_key: "f633889491eed846fb004056dd20c7c5"
server_key_iv: "7e6aa5a2df35960ec4331a6f0bc4dd29"
server_key_version: "encryptionkey@0"
I20260927 13:56:43.015010 30998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:43.037580 30998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:43.041918 30998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:43.042189 30998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:43.043159 30998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:43.044554 30998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:43.044661 30998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:43.044826 30998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:43.044898 30998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:43.077620 30998 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:46005
I20260927 13:56:43.077641 31125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:46005 every 8 connection(s)
I20260927 13:56:43.078886 30998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:56:43.082778 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 30998
I20260927 13:56:43.083009 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:56:43.083859 22635 external_mini_cluster.cc:1428] Setting key dc19a2bebbc4f26cd12a6a7cf70aedef
Sep 27 13:56:43 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517402, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:43.101574 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:43.081435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:37225 (local address 127.22.26.254:44809)
0927 13:56:43.081850 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:43.081854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:43.082952 (+ 1098us) server_negotiation.cc:407] Connection header received
0927 13:56:43.085405 (+ 2453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:43.085410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:43.085500 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:43.085618 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:43.087790 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:43.088481 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:43.090855 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:43.091300 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:43.091395 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:43.095621 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:43.095626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:43.095629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:43.095662 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:43.098300 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:43.099196 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:43.099200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:43.099202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:43.099319 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:43.099841 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:43.099844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:43.099847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:43.100192 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:43.100309 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:43.101154 (+ 845us) server_negotiation.cc:299] Negotiation successful
0927 13:56:43.101315 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":221,"threads_started":1}
I20260927 13:56:43.103053 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:43.081861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:37225)
0927 13:56:43.082751 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:43.082807 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:43.085122 (+ 2315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:43.085962 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:43.085973 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:43.086731 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:43.087587 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:43.087626 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:43.088652 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:43.088660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:43.090713 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:43.090731 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:43.091514 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:43.091523 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:43.091862 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:43.092578 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:43.092611 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:43.095394 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:43.098437 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:43.098448 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:43.098466 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:43.099057 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:43.099434 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:43.099442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:43.099448 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:43.099684 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:43.100332 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:43.100342 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:43.100779 (+ 437us) client_negotiation.cc:769] Sending connection context
0927 13:56:43.101066 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 13:56:43.101622 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":284,"threads_started":1}
I20260927 13:56:43.104622 31126 heartbeater.cc:347] Connected to a master server at 127.22.26.254:44809
I20260927 13:56:43.104949 31126 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:43.105628 31126 heartbeater.cc:511] Master 127.22.26.254:44809 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
I20260927 13:56:43.107755 30937 ts_manager.cc:194] Registered new tserver with Master: b5b110b410bd45619a27145bdd13755b (127.22.26.193:46005)
I20260927 13:56:43.109347 30937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:37225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:56:43.134588 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:44809
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:43.458947 31132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:43.459306 31132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:43.459417 31132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:43.466629 31132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:43.466735 31132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:43.466848 31132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:56:43.476600 31132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:44809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:43.477880 31132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:43.479282 31132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:43.492686 31140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:43.492873 31137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:43.492695 31138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:43.493602 31132 server_base.cc:1034] running on GCE node
I20260927 13:56:43.494272 31132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:43.495549 31132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:43.496819 31132 hybrid_clock.cc:648] HybridClock initialized: now 1790517403496767 us; error 43 us; skew 500 ppm
I20260927 13:56:43.497185 31132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:43 dist-test-slave-gkks krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517403, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:43.500360 31132 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:56:43.501950 31132 webserver.cc:460] Webserver started at http://127.22.26.194:38581/ using document root <none> and password file <none>
I20260927 13:56:43.502666 31132 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:43.502825 31132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:43.503229 31132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:43.506132 31132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "90099273c4b84de2abb9b462b4342b1e"
format_stamp: "Formatted at 2026-09-27 13:56:43 on dist-test-slave-gkks"
server_key: "f2eccb96cbcba7fb1beefee2f0d55709"
server_key_iv: "48ba5ba8829d89f5eceaa2a74ff6e2f7"
server_key_version: "encryptionkey@0"
I20260927 13:56:43.506955 31132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "90099273c4b84de2abb9b462b4342b1e"
format_stamp: "Formatted at 2026-09-27 13:56:43 on dist-test-slave-gkks"
server_key: "f2eccb96cbcba7fb1beefee2f0d55709"
server_key_iv: "48ba5ba8829d89f5eceaa2a74ff6e2f7"
server_key_version: "encryptionkey@0"
I20260927 13:56:43.512174 31132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:56:43.515755 31147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:43.517063 31132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:56:43.517256 31132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "90099273c4b84de2abb9b462b4342b1e"
format_stamp: "Formatted at 2026-09-27 13:56:43 on dist-test-slave-gkks"
server_key: "f2eccb96cbcba7fb1beefee2f0d55709"
server_key_iv: "48ba5ba8829d89f5eceaa2a74ff6e2f7"
server_key_version: "encryptionkey@0"
I20260927 13:56:43.517450 31132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:43.542696 31132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:43.547276 31132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:43.547602 31132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:43.548672 31132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:43.550117 31132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:43.550196 31132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:43.550357 31132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:43.550427 31132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:43.586753 31132 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:45231
I20260927 13:56:43.586795 31259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:45231 every 8 connection(s)
I20260927 13:56:43.588079 31132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:56:43.593369 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 31132
I20260927 13:56:43.593573 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:56:43.594107 22635 external_mini_cluster.cc:1428] Setting key d8c6e1bce1e18dd131c4d4c8daff7d23
Sep 27 13:56:43 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517403, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:43.611716 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:43.590738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:47425 (local address 127.22.26.254:44809)
0927 13:56:43.590916 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:43.590920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:43.592092 (+ 1172us) server_negotiation.cc:407] Connection header received
0927 13:56:43.594446 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:43.594451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:43.594525 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:43.594630 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:43.596498 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:43.597198 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:43.599698 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:43.600135 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:43.600236 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:43.604591 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:43.604598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:43.604604 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:43.604648 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:43.608357 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:43.609105 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:43.609113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:43.609120 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:43.609273 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:43.609709 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:43.609716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:43.609730 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:43.610123 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:43.610278 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:43.610950 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 13:56:43.611239 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:56:43.612409 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:43.591235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:47425)
0927 13:56:43.591915 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:43.591953 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:43.594108 (+ 2155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:43.594893 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:43.594902 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:43.595583 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:43.596318 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:43.596345 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:43.597364 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:43.597372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:43.599537 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:43.599555 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:43.600373 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:43.600382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:43.600693 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:43.601554 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:43.601588 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:43.604340 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:43.608505 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:43.608512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:43.608524 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:43.608959 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:43.609397 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:43.609400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:43.609403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:43.609584 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:43.610254 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:43.610259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:43.610566 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 13:56:43.610798 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 13:56:43.611260 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":260,"threads_started":1}
I20260927 13:56:43.613945 31260 heartbeater.cc:347] Connected to a master server at 127.22.26.254:44809
I20260927 13:56:43.614269 31260 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:43.615216 31260 heartbeater.cc:511] Master 127.22.26.254:44809 requested a full tablet report, sending...
I20260927 13:56:43.616827 30937 ts_manager.cc:194] Registered new tserver with Master: 90099273c4b84de2abb9b462b4342b1e (127.22.26.194:45231)
I20260927 13:56:43.617643 30937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:47425
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:56:43.648242 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:44809
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:43.977299 31265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:43.977581 31265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:43.977778 31265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:43.985002 31265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:43.985101 31265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:43.985311 31265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:56:43.995045 31265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:44809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:43.996332 31265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:43.997620 31265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:44.010483 31270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:44.010552 31271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:44.010519 31273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:44.011463 31265 server_base.cc:1034] running on GCE node
I20260927 13:56:44.012102 31265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:44.013095 31265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:44.014297 31265 hybrid_clock.cc:648] HybridClock initialized: now 1790517404014252 us; error 41 us; skew 500 ppm
I20260927 13:56:44.014667 31265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:44 dist-test-slave-gkks krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517404, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:44.017901 31265 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:56:44.019461 31265 webserver.cc:460] Webserver started at http://127.22.26.195:37711/ using document root <none> and password file <none>
I20260927 13:56:44.020182 31265 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:44.020320 31265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:44.020689 31265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:56:44.023702 31265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "2e5f3a3d0162408fae43c987cb0f3801"
format_stamp: "Formatted at 2026-09-27 13:56:44 on dist-test-slave-gkks"
server_key: "150c7ee7cf2ef10fd541e6304acb9e9a"
server_key_iv: "6506e23ef43b3fd34e8cce8d09eb59ea"
server_key_version: "encryptionkey@0"
I20260927 13:56:44.024490 31265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "2e5f3a3d0162408fae43c987cb0f3801"
format_stamp: "Formatted at 2026-09-27 13:56:44 on dist-test-slave-gkks"
server_key: "150c7ee7cf2ef10fd541e6304acb9e9a"
server_key_iv: "6506e23ef43b3fd34e8cce8d09eb59ea"
server_key_version: "encryptionkey@0"
I20260927 13:56:44.029529 31265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:56:44.032861 31280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:44.034183 31265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:56:44.034400 31265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "2e5f3a3d0162408fae43c987cb0f3801"
format_stamp: "Formatted at 2026-09-27 13:56:44 on dist-test-slave-gkks"
server_key: "150c7ee7cf2ef10fd541e6304acb9e9a"
server_key_iv: "6506e23ef43b3fd34e8cce8d09eb59ea"
server_key_version: "encryptionkey@0"
I20260927 13:56:44.034603 31265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:44.069344 31265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:44.073557 31265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:44.073879 31265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:44.074849 31265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:44.076215 31265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:44.076319 31265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:44.076478 31265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:44.076553 31265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:44.110184 31265 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:34337
I20260927 13:56:44.110270 31392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:34337 every 8 connection(s)
I20260927 13:56:44.111482 31265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:56:44.112339 31126 heartbeater.cc:503] Master 127.22.26.254:44809 was elected leader, sending a full tablet report...
I20260927 13:56:44.118628 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 31265
I20260927 13:56:44.118852 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:56:44.119375 22635 external_mini_cluster.cc:1428] Setting key 3f2654cde504db25ff6bcc1a60e1b4b0
Sep 27 13:56:44 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517404, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:56:44.131876 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:44.114236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:56437 (local address 127.22.26.254:44809)
0927 13:56:44.114720 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:44.114724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:44.115782 (+ 1058us) server_negotiation.cc:407] Connection header received
0927 13:56:44.117597 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:44.117601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:44.117662 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:44.117776 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:44.119638 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:44.120748 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:44.122637 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:44.123059 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:44.123216 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:44.126567 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:44.126574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:44.126578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:44.126608 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:44.129026 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:44.129727 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:44.129730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:44.129733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:44.129848 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:44.130215 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:44.130218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:44.130220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:44.130505 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:44.130643 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:44.131424 (+ 781us) server_negotiation.cc:299] Negotiation successful
0927 13:56:44.131584 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":259,"threads_started":1}
I20260927 13:56:44.132762 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:44.114901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:56437)
0927 13:56:44.115625 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:44.115670 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:44.117307 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:44.117974 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:44.117981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:44.118566 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:44.119423 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:44.119450 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:44.120912 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:44.120918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:44.122470 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:44.122487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:44.123318 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:44.123325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:44.123572 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:44.124175 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:44.124204 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:44.126366 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:56:44.129160 (+ 2794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:44.129166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:44.129177 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:44.129608 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:44.129946 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:56:44.129949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:56:44.129952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:56:44.130121 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:56:44.130627 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:56:44.130633 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:56:44.131026 (+ 393us) client_negotiation.cc:769] Sending connection context
0927 13:56:44.131310 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 13:56:44.131684 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":308,"threads_started":1}
I20260927 13:56:44.134297 31393 heartbeater.cc:347] Connected to a master server at 127.22.26.254:44809
I20260927 13:56:44.134598 31393 heartbeater.cc:464] Registering TS with master...
I20260927 13:56:44.135350 31393 heartbeater.cc:511] Master 127.22.26.254:44809 requested a full tablet report, sending...
I20260927 13:56:44.136682 30937 ts_manager.cc:194] Registered new tserver with Master: 2e5f3a3d0162408fae43c987cb0f3801 (127.22.26.195:34337)
I20260927 13:56:44.137483 30937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:56437
I20260927 13:56:44.146745 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab.
Entry for principal oryx/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab.
WARNING: no policy specified for oryx/127.22.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab.
Entry for principal oryx/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab.
WARNING: no policy specified for oryx/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab.
Entry for principal oryx/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab.
I20260927 13:56:44.281587 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 30998
Sep 27 13:56:44 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:56:44.299247 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:44.282914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.22.26.193:46005)
0927 13:56:44.283629 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:44.283639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:44.283673 (+ 34us) server_negotiation.cc:407] Connection header received
0927 13:56:44.283758 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:44.283764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:44.283990 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:44.284192 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:44.285619 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:44.286659 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:44.288234 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:44.288839 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:44.288998 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:44.293235 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:44.293247 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:44.293263 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:44.293298 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:44.296253 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:44.296988 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:44.296995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:44.296997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:44.297136 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:44.297529 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:44.297535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:44.297537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:44.297923 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:44.298257 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:44.298551 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 13:56:44.298828 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":356,"threads_started":1}
I20260927 13:56:44.300625 31121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:44.385020 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 30998
I20260927 13:56:44.401178 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 31132
Sep 27 13:56:44 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:56:44.419510 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:44.402218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.22.26.194:45231)
0927 13:56:44.402866 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:44.402873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:44.402900 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:56:44.402959 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:44.402965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:44.403271 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:44.403412 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:44.404818 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:44.406080 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:44.407608 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:44.408146 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:44.408255 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:44.412921 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:44.412932 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:44.412945 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:44.412977 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:44.416246 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:44.416944 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:44.416952 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:44.416955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:44.417124 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:44.417560 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:44.417565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:44.417568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:44.418240 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:44.418486 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:44.418813 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 13:56:44.419010 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":286,"threads_started":1}
I20260927 13:56:44.421123 31255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:44.513512 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 31132
I20260927 13:56:44.529775 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 31265
Sep 27 13:56:44 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:56:44.546397 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:44.530843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.22.26.195:34337)
0927 13:56:44.531505 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:44.531512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:44.531539 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:56:44.531606 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:44.531612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:44.531816 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:44.531998 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:44.533327 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:44.534287 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:44.535751 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:44.536293 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:44.536442 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:44.540775 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:44.540786 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:44.540800 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:44.540834 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:44.543553 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:44.544271 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:44.544284 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:44.544286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:44.544406 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:44.544798 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:44.544804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:44.544807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:44.545155 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:44.545462 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:44.545787 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 13:56:44.546049 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":368,"threads_started":1}
I20260927 13:56:44.547859 31388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:44.630888 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 31265
I20260927 13:56:44.646728 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 30905
I20260927 13:56:44.648177 30967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:56:44.751750 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 30905
I20260927 13:56:44.772208 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:44809
--webserver_interface=127.22.26.254
--webserver_port=34329
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:44809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:56:45.098676 31411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:45.099030 31411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:45.099228 31411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:45.106796 31411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:56:45.106945 31411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:45.107059 31411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:45.107204 31411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:56:45.107290 31411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:56:45.117388 31411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:44809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:44809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=34329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:45.118739 31411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:45.120107 31411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:45.132171 31416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:45.132270 31419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:45.132265 31417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:45.133419 31411 server_base.cc:1034] running on GCE node
I20260927 13:56:45.134186 31411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:45.135576 31411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:45.136771 31411 hybrid_clock.cc:648] HybridClock initialized: now 1790517405136714 us; error 55 us; skew 500 ppm
I20260927 13:56:45.137053 31411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:45 dist-test-slave-gkks krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517405, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:45.140110 31411 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:56:45.141503 31411 webserver.cc:460] Webserver started at http://127.22.26.254:34329/ using document root <none> and password file <none>
I20260927 13:56:45.142252 31411 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:45.142395 31411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:45.147626 31411 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:56:45.150547 31426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:45.151955 31411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:56:45.152136 31411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "6100d33e90a04a719f6f71cbbcbefbad"
format_stamp: "Formatted at 2026-09-27 13:56:42 on dist-test-slave-gkks"
server_key: "3af7f3d3029ac4cd37535e3e98dad2ff"
server_key_iv: "066015f970ede8213848c761e47b0956"
server_key_version: "encryptionkey@0"
I20260927 13:56:45.153096 31411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:45.178879 31411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:45.183041 31411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:45.183341 31411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:45.201992 31411 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:44809
I20260927 13:56:45.202005 31477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:44809 every 8 connection(s)
I20260927 13:56:45.203481 31411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:56:45.210790 31478 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:56:45.213554 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 31411
I20260927 13:56:45.214658 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:46005
--local_ip_for_outbound_sockets=127.22.26.193
--tserver_master_addrs=127.22.26.254:44809
--webserver_port=45409
--webserver_interface=127.22.26.193
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.26.193
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:56:45.214794 31478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad: Bootstrap starting.
I20260927 13:56:45.237293 31478 log.cc:824] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad: Log is configured to *not* fsync() on all Append() calls
I20260927 13:56:45.246616 31478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:56:45.247303 31478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad: Bootstrap complete.
I20260927 13:56:45.257714 31478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6100d33e90a04a719f6f71cbbcbefbad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44809 } }
I20260927 13:56:45.258699 31478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6100d33e90a04a719f6f71cbbcbefbad, State: Initialized, Role: FOLLOWER
I20260927 13:56:45.259423 31478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6100d33e90a04a719f6f71cbbcbefbad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44809 } }
I20260927 13:56:45.259687 31478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:56:45.259836 31478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:56:45.260040 31478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:56:45.264102 31478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6100d33e90a04a719f6f71cbbcbefbad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44809 } }
I20260927 13:56:45.264616 31478 leader_election.cc:302] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6100d33e90a04a719f6f71cbbcbefbad; no voters:
I20260927 13:56:45.265295 31478 leader_election.cc:288] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 13:56:45.265443 31482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:56:45.266875 31482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [term 2 LEADER]: Becoming Leader. State: Replica: 6100d33e90a04a719f6f71cbbcbefbad, State: Running, Role: LEADER
I20260927 13:56:45.267737 31482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6100d33e90a04a719f6f71cbbcbefbad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44809 } }
I20260927 13:56:45.267931 31478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:56:45.270426 31483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6100d33e90a04a719f6f71cbbcbefbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6100d33e90a04a719f6f71cbbcbefbad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44809 } } }
I20260927 13:56:45.270869 31483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:45.271111 31484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6100d33e90a04a719f6f71cbbcbefbad. Latest consensus state: current_term: 2 leader_uuid: "6100d33e90a04a719f6f71cbbcbefbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6100d33e90a04a719f6f71cbbcbefbad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 44809 } } }
I20260927 13:56:45.271521 31484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad [sys.catalog]: This master's current role is: LEADER
I20260927 13:56:45.275638 31488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:56:45.281185 31488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:56:45.282559 31488 catalog_manager.cc:1259] Loaded cluster ID: b996eaa0b2c04845a10328f57df33cea
I20260927 13:56:45.282704 31488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:56:45.286595 31488 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:56:45.288090 31488 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6100d33e90a04a719f6f71cbbcbefbad: Loaded TSK: 0
I20260927 13:56:45.289671 31488 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 13:56:45.552392 31480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:45.552703 31480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:45.552878 31480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:45.560345 31480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:45.560508 31480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:45.560679 31480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:56:45.570648 31480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.193.keytab
--principal=oryx/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:46005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=45409
--tserver_master_addrs=127.22.26.254:44809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:45.572388 31480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:45.573669 31480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:45.588368 31504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:45.588349 31505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:45.589265 31480 server_base.cc:1034] running on GCE node
W20260927 13:56:45.589538 31507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:45.590230 31480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:45.591235 31480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:45.592461 31480 hybrid_clock.cc:648] HybridClock initialized: now 1790517405592397 us; error 58 us; skew 500 ppm
I20260927 13:56:45.592788 31480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:45 dist-test-slave-gkks krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517405, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:45.596055 31480 init.cc:377] Logged in from keytab as oryx/127.22.26.193@KRBTEST.COM (short username oryx)
I20260927 13:56:45.597656 31480 webserver.cc:460] Webserver started at http://127.22.26.193:45409/ using document root <none> and password file <none>
I20260927 13:56:45.598356 31480 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:45.598515 31480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:45.604051 31480 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 13:56:45.607070 31514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:45.608621 31480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:56:45.608790 31480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "b5b110b410bd45619a27145bdd13755b"
format_stamp: "Formatted at 2026-09-27 13:56:42 on dist-test-slave-gkks"
server_key: "f633889491eed846fb004056dd20c7c5"
server_key_iv: "7e6aa5a2df35960ec4331a6f0bc4dd29"
server_key_version: "encryptionkey@0"
I20260927 13:56:45.609740 31480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:45.624949 31480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:45.629372 31480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:45.629662 31480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:45.630658 31480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:45.632089 31480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:45.632174 31480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:45.632329 31480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:45.632427 31480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:45.666139 31480 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:46005
I20260927 13:56:45.666236 31626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:46005 every 8 connection(s)
I20260927 13:56:45.667673 31480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:56:45.669265 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 31480
I20260927 13:56:45.669951 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:45231
--local_ip_for_outbound_sockets=127.22.26.194
--tserver_master_addrs=127.22.26.254:44809
--webserver_port=38581
--webserver_interface=127.22.26.194
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.26.194
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:56:45 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:45 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:56:45.999308 31631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:45.999621 31631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:45.999784 31631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:46.007167 31631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:46.007299 31631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:46.007470 31631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:56:46.017292 31631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.194.keytab
--principal=oryx/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:45231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=38581
--tserver_master_addrs=127.22.26.254:44809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:46.018643 31631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:46.019994 31631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:46.038105 31637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:46.038239 31638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:46.039603 31640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:46.039551 31631 server_base.cc:1034] running on GCE node
I20260927 13:56:46.040694 31631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:46.042412 31631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:46.043733 31631 hybrid_clock.cc:648] HybridClock initialized: now 1790517406043612 us; error 89 us; skew 500 ppm
I20260927 13:56:46.044309 31631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:46 dist-test-slave-gkks krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517406, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:46.049832 31631 init.cc:377] Logged in from keytab as oryx/127.22.26.194@KRBTEST.COM (short username oryx)
I20260927 13:56:46.052287 31631 webserver.cc:460] Webserver started at http://127.22.26.194:38581/ using document root <none> and password file <none>
I20260927 13:56:46.053550 31631 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:46.053761 31631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:46.063475 31631 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 13:56:46.066475 31647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:46.068337 31631 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260927 13:56:46.068548 31631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "90099273c4b84de2abb9b462b4342b1e"
format_stamp: "Formatted at 2026-09-27 13:56:43 on dist-test-slave-gkks"
server_key: "f2eccb96cbcba7fb1beefee2f0d55709"
server_key_iv: "48ba5ba8829d89f5eceaa2a74ff6e2f7"
server_key_version: "encryptionkey@0"
I20260927 13:56:46.069772 31631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:46.100785 31631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:46.105216 31631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:46.105494 31631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:46.106706 31631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:46.108239 31631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:46.108346 31631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:46.108502 31631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:46.108570 31631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:46.150259 31631 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:45231
I20260927 13:56:46.150312 31759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:45231 every 8 connection(s)
I20260927 13:56:46.151824 31631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:56:46.154233 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 31631
I20260927 13:56:46.155400 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:34337
--local_ip_for_outbound_sockets=127.22.26.195
--tserver_master_addrs=127.22.26.254:44809
--webserver_port=37711
--webserver_interface=127.22.26.195
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.26.195
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:56:46 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:46 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:56:46.807618 31630 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.121s user 0.004s sys 0.014s
I20260927 13:56:46.809744 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:45.671037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:52577)
0927 13:56:45.672201 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:45.672247 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:45.673936 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:45.675570 (+ 1634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:45.675579 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:45.676095 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:45.676788 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:45.676807 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:45.678586 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:45.678590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:45.679945 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:45.679957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:45.681009 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:45.681016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:45.681277 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:45.687047 (+ 5770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:45.687070 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:46.808583 (+1121513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":661,"threads_started":1}
W20260927 13:56:46.809791 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:45.671015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:52577 (local address 127.22.26.254:44809)
0927 13:56:45.671800 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:45.671819 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:45.672386 (+ 567us) server_negotiation.cc:407] Connection header received
0927 13:56:45.674550 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:45.674571 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:45.675011 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:45.675457 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:45.676997 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:45.678427 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:45.680109 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:45.680772 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:45.680976 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:46.809003 (+1128027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:52577: BlockingRecv error: recv got EOF from 127.22.26.193:52577 (error 108)
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":257,"threads_started":1}
W20260927 13:56:46.810935 31627 heartbeater.cc:644] Failed to heartbeat to 127.22.26.254:44809 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.26.254:44809: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Sep 27 13:56:46 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:46 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:46.820986 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:46.811919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:51011)
0927 13:56:46.812375 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:46.812389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:46.812535 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:46.812869 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:46.812873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:46.813348 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:46.813509 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:46.813517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:46.814643 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:46.814649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:46.815652 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:46.815663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:46.816353 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:46.816359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:46.816527 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:46.817143 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:46.817160 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:46.820703 (+ 3543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251}
W20260927 13:56:46.821131 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:46.812074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:51011 (local address 127.22.26.254:44809)
0927 13:56:46.812252 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:46.812258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:46.812465 (+ 207us) server_negotiation.cc:407] Connection header received
0927 13:56:46.812668 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:46.812672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:46.812738 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:46.812840 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:46.813654 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:46.814508 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:46.815794 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:46.816179 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:46.816321 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:46.820892 (+ 4571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:51011: BlockingRecv error: recv got EOF from 127.22.26.193:51011 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 13:56:46 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:46 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:56:46.830300 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:46.822020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:49561 (local address 127.22.26.254:44809)
0927 13:56:46.822161 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:46.822165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:46.822357 (+ 192us) server_negotiation.cc:407] Connection header received
0927 13:56:46.822616 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:46.822621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:46.822698 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:46.822805 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:46.823487 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:46.824181 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:46.825385 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:46.825803 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:46.825937 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:46.830065 (+ 4128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:49561: BlockingRecv error: recv got EOF from 127.22.26.193:49561 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 13:56:46.830660 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:46.821967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:49561)
0927 13:56:46.822271 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:46.822284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:46.822447 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:46.822836 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:46.822840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:46.823205 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:46.823356 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:46.823363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:46.824328 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:46.824333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:46.825249 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:46.825260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:46.825950 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:46.825955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:46.826128 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:46.826721 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:46.826738 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:46.829884 (+ 3146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210}
W20260927 13:56:46.830946 31627 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 13:56:46.850219 31764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:56:46.850782 31764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:56:46.851020 31764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:56:46.869869 31764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:56:46.870083 31764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:56:46.870379 31764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:56:46.881645 31764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/krb5kdc/oryx_127.22.26.195.keytab
--principal=oryx/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:34337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=37711
--tserver_master_addrs=127.22.26.254:44809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:56:46.883074 31764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:56:46.884518 31764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:56:46.901103 31770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:46.900988 31771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:56:46.901722 31773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:56:46.903762 31764 server_base.cc:1034] running on GCE node
I20260927 13:56:46.904475 31764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:56:46.905519 31764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:56:46.906906 31764 hybrid_clock.cc:648] HybridClock initialized: now 1790517406906831 us; error 85 us; skew 500 ppm
I20260927 13:56:46.907295 31764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:56:46 dist-test-slave-gkks krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517406, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:56:46.910560 31764 init.cc:377] Logged in from keytab as oryx/127.22.26.195@KRBTEST.COM (short username oryx)
I20260927 13:56:46.912226 31764 webserver.cc:460] Webserver started at http://127.22.26.195:37711/ using document root <none> and password file <none>
I20260927 13:56:46.913026 31764 fs_manager.cc:359] Metadata directory not provided
I20260927 13:56:46.913192 31764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:56:46.918761 31764 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:56:46.921880 31780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:46.923682 31764 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 13:56:46.923869 31764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "2e5f3a3d0162408fae43c987cb0f3801"
format_stamp: "Formatted at 2026-09-27 13:56:44 on dist-test-slave-gkks"
server_key: "150c7ee7cf2ef10fd541e6304acb9e9a"
server_key_iv: "6506e23ef43b3fd34e8cce8d09eb59ea"
server_key_version: "encryptionkey@0"
I20260927 13:56:46.924839 31764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:56:46.944442 31764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:56:46.949071 31764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:56:46.949335 31764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:56:46.950394 31764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:56:46.951877 31764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:56:46.951997 31764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:46.952137 31764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:56:46.952224 31764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:56:47.003124 31764 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:34337
I20260927 13:56:47.003170 31892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:34337 every 8 connection(s)
I20260927 13:56:47.004616 31764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:56:47.005957 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 31764
Sep 27 13:56:47 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:47 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:47.046377 31629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:47.007418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.22.26.254:44809)
0927 13:56:47.007702 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:47.007713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:47.024359 (+ 16646us) server_negotiation.cc:407] Connection header received
0927 13:56:47.024603 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:47.024608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:47.024675 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:47.024775 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:47.027663 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.028721 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.031821 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.032347 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.032482 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:47.037241 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:56:47.037254 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:56:47.037280 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:56:47.037338 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:56:47.041763 (+ 4425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:47.042967 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:47.042978 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:47.042990 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:47.043240 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:56:47.043776 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:56:47.043784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:56:47.043790 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:56:47.044499 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:56:47.044855 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0927 13:56:47.045467 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 13:56:47.045909 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260927 13:56:47.475853 31763 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.293s user 0.009s sys 0.009s
W20260927 13:56:47.477167 31762 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:46.154540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:58471 (local address 127.22.26.254:44809)
0927 13:56:46.169915 (+ 15375us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:46.169922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:46.170385 (+ 463us) server_negotiation.cc:407] Connection header received
0927 13:56:46.172741 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:46.172751 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:46.172914 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:46.173181 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:46.175583 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:46.177382 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:46.180207 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:46.181200 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:46.181391 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:47.476834 (+1295443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:58471: BlockingRecv error: recv got EOF from 127.22.26.194:58471 (error 108)
Metrics: {"server-negotiator.queue_time_us":15246,"thread_start_us":15366,"threads_started":1}
I20260927 13:56:47.477799 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:46.155384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:58471)
0927 13:56:46.170121 (+ 14737us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:46.170198 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:46.172418 (+ 2220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:46.173405 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:46.173421 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:46.174411 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:46.175340 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:46.175371 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:46.177657 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:46.177665 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:46.180010 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:46.180029 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:46.181483 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:46.181492 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:46.181843 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:46.183110 (+ 1267us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:46.183151 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:47.476625 (+1293474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13641,"thread_start_us":13299,"threads_started":1}
W20260927 13:56:47.478752 31760 heartbeater.cc:644] Failed to heartbeat to 127.22.26.254:44809 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.26.254:44809: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Sep 27 13:56:47 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:47 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:47.491791 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:47.479584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:43533)
0927 13:56:47.479938 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:47.479966 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:47.480140 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:47.480513 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:47.480521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:47.481193 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:47.481514 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:47.481530 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.482621 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.482629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:47.484893 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:47.484913 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.485664 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.485674 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:47.485936 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:47.486729 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:47.486764 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:47.491495 (+ 4731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182}
W20260927 13:56:47.491969 31762 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:47.479564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:43533 (local address 127.22.26.254:44809)
0927 13:56:47.479717 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:47.479721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:47.480057 (+ 336us) server_negotiation.cc:407] Connection header received
0927 13:56:47.480284 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:47.480288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:47.480350 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:47.480430 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:47.481690 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.482469 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.485060 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.485455 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.485550 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:47.491731 (+ 6181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:43533: BlockingRecv error: recv got EOF from 127.22.26.194:43533 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 13:56:47 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:47 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:47.502497 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:47.492542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:49091)
0927 13:56:47.492781 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:47.492810 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:47.492962 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:47.493311 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:47.493318 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:47.493915 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:47.494210 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:47.494227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.495418 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.495423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:47.496602 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:47.496615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.497501 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.497507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:47.497698 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:47.498290 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:47.498307 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:47.502177 (+ 3870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 13:56:47.502839 31760 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 13:56:47.503060 31762 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:47.492699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:49091 (local address 127.22.26.254:44809)
0927 13:56:47.492847 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:47.492851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:47.492866 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:56:47.493090 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:47.493093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:47.493160 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:47.493237 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:47.494378 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.495271 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.496742 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.497344 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.497459 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:47.502378 (+ 4919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:49091: BlockingRecv error: recv got EOF from 127.22.26.194:49091 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 13:56:47 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:47 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:47.842752 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:47.831617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:51811)
0927 13:56:47.832193 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:47.832209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:47.832330 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:47.832833 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:47.832836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:47.833236 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:47.833378 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:47.833385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.835364 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.835368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:47.836358 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:47.836371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.837787 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.837791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:47.837934 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:47.838676 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:47.838701 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:47.842488 (+ 3787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":322,"threads_started":1}
W20260927 13:56:47.843350 31762 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:47.831748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:51811 (local address 127.22.26.254:44809)
0927 13:56:47.832017 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:47.832027 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:47.832306 (+ 279us) server_negotiation.cc:407] Connection header received
0927 13:56:47.832493 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:47.832501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:47.832645 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:47.832790 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:47.833538 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.835162 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.836520 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.837584 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.837761 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:47.842961 (+ 5200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:51811: BlockingRecv error: recv got EOF from 127.22.26.193:51811 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260927 13:56:48.165863 31896 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.135s user 0.003s sys 0.013s
W20260927 13:56:48.167049 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:47.014826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:38089 (local address 127.22.26.254:44809)
0927 13:56:47.023700 (+ 8874us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:47.023709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:47.023737 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:56:47.023806 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:47.023813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:47.023898 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:47.024015 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:47.025935 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.026732 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.028952 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.029434 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.029582 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:48.166818 (+1137236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:38089: BlockingRecv error: recv got EOF from 127.22.26.195:38089 (error 108)
Metrics: {"server-negotiator.queue_time_us":8670,"thread_start_us":8498,"threads_started":1}
I20260927 13:56:48.167629 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:47.013834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:38089)
0927 13:56:47.015040 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:47.015137 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:47.018963 (+ 3826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:47.024376 (+ 5413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:47.024392 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:47.025152 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:47.025762 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:47.025780 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.026919 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.026924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:47.028810 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:47.028824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:47.029687 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:47.029696 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:47.029933 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:47.030872 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:47.030899 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:48.166626 (+1135727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":979,"threads_started":1}
W20260927 13:56:48.168537 31893 heartbeater.cc:644] Failed to heartbeat to 127.22.26.254:44809 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.26.254:44809: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Sep 27 13:56:48 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:48 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:48.178304 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:48.169242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:54103)
0927 13:56:48.169485 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:48.169506 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:48.169619 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:48.169969 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:48.169974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:48.170467 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:48.170695 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:48.170713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.171729 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.171732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:48.172757 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:48.172770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.173489 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.173497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:48.173664 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:48.174303 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:48.174319 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:48.178091 (+ 3772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260927 13:56:48.178486 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:48.169369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:54103 (local address 127.22.26.254:44809)
0927 13:56:48.169664 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:48.169671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:48.169693 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:56:48.169752 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:48.169755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:48.169811 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:48.169902 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:48.170832 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.171609 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.172875 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.173255 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.173345 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:48.178295 (+ 4950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:54103: BlockingRecv error: recv got EOF from 127.22.26.195:54103 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Sep 27 13:56:48 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:48 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:48.187443 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:48.179238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:51749)
0927 13:56:48.179397 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:48.179420 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:48.179499 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:48.179776 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:48.179780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:48.180038 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:48.180187 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:48.180194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.181150 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.181158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:48.182345 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:48.182357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.183106 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.183113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:48.183284 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:48.183779 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:48.183795 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:48.187159 (+ 3364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 13:56:48.187661 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:48.179259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:51749 (local address 127.22.26.254:44809)
0927 13:56:48.179442 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:48.179446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:48.179462 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:56:48.179592 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:48.179596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:48.179648 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:48.179738 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:48.180297 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.181005 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.182493 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.182931 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.183071 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:48.187377 (+ 4306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:51749: BlockingRecv error: recv got EOF from 127.22.26.195:51749 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 13:56:48.187885 31893 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 13:56:48 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:48 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:48.512987 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:48.503560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:39721)
0927 13:56:48.504154 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:48.504171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:48.504335 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:48.504698 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:48.504705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:48.505233 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:48.505415 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:48.505429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.506677 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.506681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:48.507683 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:48.507694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.508415 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.508419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:48.508537 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:48.509141 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:48.509158 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:48.512742 (+ 3584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":333,"threads_started":1}
W20260927 13:56:48.513146 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:48.503635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:39721 (local address 127.22.26.254:44809)
0927 13:56:48.503927 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:48.503935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:48.504267 (+ 332us) server_negotiation.cc:407] Connection header received
0927 13:56:48.504467 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:48.504471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:48.504533 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:48.504638 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:48.505579 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.506494 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.507814 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.508225 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.508342 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:48.512952 (+ 4610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:39721: BlockingRecv error: recv got EOF from 127.22.26.194:39721 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 27 13:56:48 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:48 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:48.852599 31901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:48.843757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:47117)
0927 13:56:48.844234 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:48.844251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:48.844366 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:48.844652 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:48.844655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:48.845054 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:48.845232 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:48.845240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.846248 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.846252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:48.847200 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:48.847212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.847896 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.847900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:48.848024 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:48.848583 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:48.848600 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:48.852351 (+ 3751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":265,"threads_started":1}
W20260927 13:56:48.852753 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:48.843913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:47117 (local address 127.22.26.254:44809)
0927 13:56:48.844162 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:48.844168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:48.844329 (+ 161us) server_negotiation.cc:407] Connection header received
0927 13:56:48.844466 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:48.844469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:48.844528 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:48.844606 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:48.845355 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.846103 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.847334 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:48.847756 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:48.847859 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:48.852535 (+ 4676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:47117: BlockingRecv error: recv got EOF from 127.22.26.193:47117 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 13:56:49 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:49 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:49.197192 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:49.188504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:49083)
0927 13:56:49.189097 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:49.189114 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:49.189225 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:49.189533 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:49.189537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:49.189966 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:49.190112 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:49.190120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:49.191056 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:49.191059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:49.192062 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:49.192074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:49.192951 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:49.192956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:49.193074 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:49.193677 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:49.193693 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:49.196960 (+ 3267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":342,"threads_started":1}
W20260927 13:56:49.197424 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:49.188671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:49083 (local address 127.22.26.254:44809)
0927 13:56:49.188875 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:49.188879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:49.189193 (+ 314us) server_negotiation.cc:407] Connection header received
0927 13:56:49.189322 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:49.189325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:49.189384 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:49.189507 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:49.190242 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:49.190944 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:49.192226 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:49.192785 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:49.192897 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:49.197177 (+ 4280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:49083: BlockingRecv error: recv got EOF from 127.22.26.195:49083 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 13:56:49 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:49 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:49.523713 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:49.513942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:41277)
0927 13:56:49.514552 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:49.514569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:49.514692 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:49.515042 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:49.515046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:49.515573 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:49.515784 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:49.515795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:49.516756 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:49.516760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:49.517734 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:49.517747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:49.518446 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:49.518451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:49.518600 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:49.519294 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:49.519314 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:49.523369 (+ 4055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":391,"threads_started":1}
W20260927 13:56:49.523736 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:49.513978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:41277 (local address 127.22.26.254:44809)
0927 13:56:49.514179 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:49.514188 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:49.514649 (+ 461us) server_negotiation.cc:407] Connection header received
0927 13:56:49.514792 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:49.514796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:49.514856 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:49.514939 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:49.515914 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:49.516628 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:49.517885 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:49.518299 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:49.518400 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:49.523428 (+ 5028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:41277: BlockingRecv error: recv got EOF from 127.22.26.194:41277 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 13:56:49 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:49 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:49.862651 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:49.853619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:58441)
0927 13:56:49.854220 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:49.854240 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:49.854390 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:49.854746 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:49.854750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:49.855294 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:49.855532 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:49.855543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:49.856565 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:49.856569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:49.857596 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:49.857612 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:49.858325 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:49.858329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:49.858445 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:49.859010 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:49.859026 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:49.862411 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":312,"threads_started":1}
W20260927 13:56:49.862823 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:49.853790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:58441 (local address 127.22.26.254:44809)
0927 13:56:49.854046 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:49.854053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:49.854310 (+ 257us) server_negotiation.cc:407] Connection header received
0927 13:56:49.854527 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:49.854531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:49.854592 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:49.854666 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:49.855664 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:49.856391 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:49.857782 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:49.858208 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:49.858303 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:49.862606 (+ 4303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:58441: BlockingRecv error: recv got EOF from 127.22.26.193:58441 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 13:56:50 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:50 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:50.207926 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:50.198266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:53749)
0927 13:56:50.199419 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:50.199436 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:50.199556 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:50.199914 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:50.199922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:50.200347 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:50.200526 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:50.200536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:50.201565 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:50.201571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:50.202565 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:50.202577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:50.203386 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:50.203392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:50.203589 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:50.204241 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:50.204258 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:50.207681 (+ 3423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":887,"threads_started":1}
W20260927 13:56:50.208158 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:50.198476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:53749 (local address 127.22.26.254:44809)
0927 13:56:50.198702 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:50.198709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:50.199518 (+ 809us) server_negotiation.cc:407] Connection header received
0927 13:56:50.199677 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:50.199682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:50.199763 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:50.199874 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:50.200685 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:50.201416 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:50.202719 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:50.203221 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:50.203374 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:50.207900 (+ 4526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:53749: BlockingRecv error: recv got EOF from 127.22.26.195:53749 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 13:56:50 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:50 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:50.533605 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:50.524731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:46495)
0927 13:56:50.525283 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:50.525300 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:50.525417 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:50.525711 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:50.525715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:50.526047 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:50.526238 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:50.526246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:50.527386 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:50.527390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:50.528423 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:50.528435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:50.529074 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:50.529078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:50.529200 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:50.529827 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:50.529844 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:50.533350 (+ 3506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":283,"threads_started":1}
W20260927 13:56:50.533819 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:50.524763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:46495 (local address 127.22.26.254:44809)
0927 13:56:50.524943 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:50.524947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:50.525434 (+ 487us) server_negotiation.cc:407] Connection header received
0927 13:56:50.525511 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:50.525514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:50.525579 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:50.525668 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:50.526370 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:50.527255 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:50.528549 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:50.528949 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:50.529043 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:50.533611 (+ 4568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:46495: BlockingRecv error: recv got EOF from 127.22.26.194:46495 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 13:56:50 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:50 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:50.878259 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:50.863741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:55239)
0927 13:56:50.864326 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:50.864342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:50.864493 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:50.864873 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:50.864878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:50.865302 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:50.865480 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:50.865489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:50.866551 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:50.866559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:50.867589 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:50.867600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:50.868328 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:50.868333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:50.868462 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:50.873920 (+ 5458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:50.873940 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:50.877993 (+ 4053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":293,"threads_started":1}
W20260927 13:56:50.878525 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:50.863886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:55239 (local address 127.22.26.254:44809)
0927 13:56:50.864119 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:50.864127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:50.864433 (+ 306us) server_negotiation.cc:407] Connection header received
0927 13:56:50.864660 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:50.864664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:50.864727 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:50.864835 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:50.865680 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:50.866415 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:50.867727 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:50.868137 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:50.868303 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:50.878256 (+ 9953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:55239: BlockingRecv error: recv got EOF from 127.22.26.193:55239 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 13:56:51 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:51 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:51.218000 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:51.208956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:51387)
0927 13:56:51.209499 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:51.209515 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:51.209708 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:51.210040 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:51.210046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:51.210413 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:51.210592 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:51.210602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:51.211679 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:51.211685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:51.212620 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:51.212634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:51.213322 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:51.213328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:51.213497 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:51.214100 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:51.214116 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:51.217743 (+ 3627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":287,"threads_started":1}
W20260927 13:56:51.218259 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:51.209038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:51387 (local address 127.22.26.254:44809)
0927 13:56:51.209266 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:51.209275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:51.209631 (+ 356us) server_negotiation.cc:407] Connection header received
0927 13:56:51.209828 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:51.209833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:51.209901 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:51.210004 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:51.210738 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:51.211546 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:51.212777 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:51.213173 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:51.213299 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:51.217905 (+ 4606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:51387: BlockingRecv error: recv got EOF from 127.22.26.195:51387 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 13:56:51 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:51 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:51.558322 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:51.534702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:35121)
0927 13:56:51.535266 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:51.535282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:51.535437 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:51.535795 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:51.535800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:51.536207 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:51.536371 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:51.536379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:51.537432 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:51.537439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:51.538429 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:51.538439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:51.539171 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:51.539176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:51.539359 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:51.554144 (+ 14785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:51.554173 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:51.558030 (+ 3857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":301,"threads_started":1}
W20260927 13:56:51.558446 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:51.534776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:35121 (local address 127.22.26.254:44809)
0927 13:56:51.535043 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:51.535067 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:51.535373 (+ 306us) server_negotiation.cc:407] Connection header received
0927 13:56:51.535563 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:51.535568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:51.535653 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:51.535761 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:51.536517 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:51.537293 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:51.538568 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:51.538993 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:51.539159 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:51.558220 (+ 19061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:35121: BlockingRecv error: recv got EOF from 127.22.26.194:35121 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 27 13:56:51 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:51 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:51.888813 31910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:51.879237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:58821)
0927 13:56:51.879879 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:51.879894 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:51.880057 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:51.880408 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:51.880413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:51.880867 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:51.881010 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:51.881019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:51.882035 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:51.882042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:51.882983 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:51.882994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:51.883729 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:51.883734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:51.883922 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:51.884602 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:51.884619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:51.888515 (+ 3896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":398,"threads_started":1}
W20260927 13:56:51.889010 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:51.879315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:58821 (local address 127.22.26.254:44809)
0927 13:56:51.879683 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:51.879691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:51.879987 (+ 296us) server_negotiation.cc:407] Connection header received
0927 13:56:51.880191 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:51.880197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:51.880267 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:51.880373 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:51.881179 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:51.881893 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:51.883150 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:51.883579 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:51.883717 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:51.888747 (+ 5030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:58821: BlockingRecv error: recv got EOF from 127.22.26.193:58821 (error 108)
Metrics: {"server-negotiator.queue_time_us":204}
Sep 27 13:56:52 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:52 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:52.228300 31911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:52.218906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:37663)
0927 13:56:52.219473 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:52.219489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:52.219594 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:52.219910 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:52.219914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:52.220292 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:52.220506 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:52.220518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:52.222050 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:52.222057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:52.223172 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:52.223193 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:52.223875 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:52.223879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:52.224001 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:52.224565 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:52.224582 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:52.227980 (+ 3398us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":329,"threads_started":1}
W20260927 13:56:52.228442 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:52.219038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:37663 (local address 127.22.26.254:44809)
0927 13:56:52.219259 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:52.219263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:52.219568 (+ 305us) server_negotiation.cc:407] Connection header received
0927 13:56:52.219709 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:52.219713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:52.219778 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:52.219866 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:52.220696 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:52.221899 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:52.223311 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:52.223750 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:52.223851 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:52.228178 (+ 4327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:37663: BlockingRecv error: recv got EOF from 127.22.26.195:37663 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 27 13:56:52 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:52 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:52.568094 31912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:52.559266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:32837)
0927 13:56:52.559815 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:52.559831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:52.559988 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:52.560339 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:52.560344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:52.560778 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:52.560920 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:52.560928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:52.561928 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:52.561936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:52.562879 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:52.562889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:52.563646 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:52.563651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:52.563827 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:52.564452 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:52.564469 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:52.567779 (+ 3310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":305,"threads_started":1}
W20260927 13:56:52.568210 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:52.559466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:32837 (local address 127.22.26.254:44809)
0927 13:56:52.559713 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:52.559719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:52.559919 (+ 200us) server_negotiation.cc:407] Connection header received
0927 13:56:52.560115 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:52.560121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:52.560189 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:52.560307 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:52.561070 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:52.561796 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:52.563026 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:52.563502 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:52.563635 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:52.567971 (+ 4336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:32837: BlockingRecv error: recv got EOF from 127.22.26.194:32837 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 13:56:52 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:52 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:52.899381 31913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:52.889840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:43283)
0927 13:56:52.890312 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:52.890328 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:52.890443 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:52.890759 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:52.890762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:52.891263 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:52.891440 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:52.891450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:52.892507 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:52.892511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:52.893430 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:52.893441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:52.894238 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:52.894243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:52.894421 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:52.895215 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:52.895238 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:52.899123 (+ 3885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":255,"threads_started":1}
W20260927 13:56:52.899675 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:52.889948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:43283 (local address 127.22.26.254:44809)
0927 13:56:52.890196 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:52.890203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:52.890408 (+ 205us) server_negotiation.cc:407] Connection header received
0927 13:56:52.890563 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:52.890567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:52.890629 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:52.890711 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:52.891615 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:52.892381 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:52.893563 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:52.894053 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:52.894233 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:52.899415 (+ 5182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:43283: BlockingRecv error: recv got EOF from 127.22.26.193:43283 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 13:56:53 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:53 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:53.237608 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:53.229250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:33239)
0927 13:56:53.229798 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:53.229824 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:53.229936 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:53.230362 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:53.230366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:53.230797 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:53.230951 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:53.230958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:53.232005 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:53.232009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:53.232948 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:53.232960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:53.233593 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:53.233597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:53.233715 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:53.234281 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:53.234297 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:53.237393 (+ 3096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":285,"threads_started":1}
W20260927 13:56:53.237874 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:53.229285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:33239 (local address 127.22.26.254:44809)
0927 13:56:53.229509 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:53.229517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:53.229908 (+ 391us) server_negotiation.cc:407] Connection header received
0927 13:56:53.230117 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:53.230121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:53.230186 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:53.230340 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:53.231113 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:53.231885 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:53.233074 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:53.233463 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:53.233560 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:53.237622 (+ 4062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:33239: BlockingRecv error: recv got EOF from 127.22.26.195:33239 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 13:56:53 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:53 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:53.578557 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:53.569129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:47101)
0927 13:56:53.569591 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:53.569607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:53.569738 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:53.570021 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:53.570024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:53.570415 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:53.570583 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:53.570591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:53.571724 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:53.571728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:53.572771 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:53.572783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:53.573736 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:53.573761 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:53.573972 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:53.574598 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:53.574615 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:53.578276 (+ 3661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":269,"threads_started":1}
W20260927 13:56:53.578706 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:53.569161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:47101 (local address 127.22.26.254:44809)
0927 13:56:53.569420 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:53.569427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:53.569691 (+ 264us) server_negotiation.cc:407] Connection header received
0927 13:56:53.569838 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:53.569842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:53.569903 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:53.569983 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:53.570736 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:53.571559 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:53.572901 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:53.573545 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:53.573690 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:53.578480 (+ 4790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:47101: BlockingRecv error: recv got EOF from 127.22.26.194:47101 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 13:56:53 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:53 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:53.909560 31916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:53.900377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:36543)
0927 13:56:53.900968 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:53.900984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:53.901107 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:53.901449 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:53.901452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:53.901870 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:53.902043 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:53.902051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:53.903008 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:53.903011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:53.903990 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:53.904002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:53.904653 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:53.904658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:53.904794 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:53.905361 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:53.905383 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:53.909319 (+ 3936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":284,"threads_started":1}
W20260927 13:56:53.909777 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:53.900450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:36543 (local address 127.22.26.254:44809)
0927 13:56:53.900690 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:53.900694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:53.901092 (+ 398us) server_negotiation.cc:407] Connection header received
0927 13:56:53.901205 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:53.901209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:53.901274 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:53.901367 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:53.902173 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:53.902892 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:53.904124 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:53.904517 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:53.904612 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:53.909527 (+ 4915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:36543: BlockingRecv error: recv got EOF from 127.22.26.193:36543 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 27 13:56:54 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:54 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:54.247335 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:54.238598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:46679)
0927 13:56:54.239068 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:54.239102 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:54.239223 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:54.239518 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:54.239522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:54.239918 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:54.240082 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:54.240090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:54.241102 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:54.241106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:54.242012 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:54.242023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:54.242834 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:54.242838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:54.242961 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:54.243556 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:54.243573 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:54.247040 (+ 3467us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":264,"threads_started":1}
W20260927 13:56:54.247627 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:54.238714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:46679 (local address 127.22.26.254:44809)
0927 13:56:54.238971 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:54.238978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:54.239171 (+ 193us) server_negotiation.cc:407] Connection header received
0927 13:56:54.239328 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:54.239331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:54.239396 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:54.239478 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:54.240202 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:54.240983 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:54.242179 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:54.242689 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:54.242802 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:54.247312 (+ 4510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:46679: BlockingRecv error: recv got EOF from 127.22.26.195:46679 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 27 13:56:54 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:54 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:54.589049 31918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:54.579579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:55095)
0927 13:56:54.580220 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:54.580237 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:54.580347 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:54.580641 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:54.580645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:54.581072 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:54.581240 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:54.581248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:54.582304 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:54.582308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:54.583282 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:54.583294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:54.584019 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:54.584024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:54.584144 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:54.584830 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:54.584850 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:54.588674 (+ 3824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":319,"threads_started":1}
W20260927 13:56:54.589082 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:54.579649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:55095 (local address 127.22.26.254:44809)
0927 13:56:54.579866 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:54.579870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:54.580311 (+ 441us) server_negotiation.cc:407] Connection header received
0927 13:56:54.580444 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:54.580448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:54.580509 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:54.580592 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:54.581396 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:54.582164 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:54.583421 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:54.583849 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:54.583950 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:54.588882 (+ 4932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:55095: BlockingRecv error: recv got EOF from 127.22.26.194:55095 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 13:56:54 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:54 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:54.920106 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:54.910603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:52399)
0927 13:56:54.911141 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:54.911158 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:54.911292 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:54.911578 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:54.911581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:54.912151 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:54.912401 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:54.912412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:54.913703 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:54.913709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:54.914699 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:54.914711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:54.915543 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:54.915549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:54.915711 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:54.916288 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:54.916304 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:54.919755 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":285,"threads_started":1}
W20260927 13:56:54.920212 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:54.910721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:52399 (local address 127.22.26.254:44809)
0927 13:56:54.910977 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:54.910983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:54.911249 (+ 266us) server_negotiation.cc:407] Connection header received
0927 13:56:54.911389 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:54.911393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:54.911456 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:54.911539 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:54.912556 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:54.913563 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:54.914842 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:54.915357 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:54.915485 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:54.919958 (+ 4473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:52399: BlockingRecv error: recv got EOF from 127.22.26.193:52399 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 13:56:55 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:55 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:55.257386 31920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:55.248471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:46777)
0927 13:56:55.249054 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:55.249070 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:55.249194 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:55.249484 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:55.249487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:55.249879 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:55.250046 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:55.250054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:55.251022 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:55.251026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:55.252049 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:55.252061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:55.252948 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:55.252952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:55.253064 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:55.253618 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:55.253635 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:55.257042 (+ 3407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":287,"threads_started":1}
W20260927 13:56:55.257468 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:55.248612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:46777 (local address 127.22.26.254:44809)
0927 13:56:55.248868 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:55.248874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:55.249171 (+ 297us) server_negotiation.cc:407] Connection header received
0927 13:56:55.249296 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:55.249300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:55.249364 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:55.249449 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:55.250177 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:55.250892 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:55.252193 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:55.252804 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:55.252919 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:55.257246 (+ 4327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:46777: BlockingRecv error: recv got EOF from 127.22.26.195:46777 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 13:56:55 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:55 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:55.599529 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:55.590379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:44461)
0927 13:56:55.590971 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:55.590987 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:55.591168 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:55.591567 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:55.591575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:55.591996 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:55.592202 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:55.592211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:55.593350 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:55.593354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:55.594312 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:55.594323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:55.595013 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:55.595017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:55.595164 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:55.595742 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:55.595758 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:55.599157 (+ 3399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":332,"threads_started":1}
W20260927 13:56:55.599637 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:55.590438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:44461 (local address 127.22.26.254:44809)
0927 13:56:55.590695 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:55.590701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:55.591044 (+ 343us) server_negotiation.cc:407] Connection header received
0927 13:56:55.591315 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:55.591320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:55.591401 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:55.591501 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:55.592326 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:55.593224 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:55.594434 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:55.594854 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:55.594945 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:55.599401 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:44461: BlockingRecv error: recv got EOF from 127.22.26.194:44461 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 13:56:55 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:55 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:55.941998 31922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:55.921058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:48777)
0927 13:56:55.921503 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:55.921519 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:55.921627 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:55.921903 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:55.921906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:55.922294 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:55.922435 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:55.922443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:55.923420 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:55.923423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:55.924357 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:55.924369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:55.925107 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:55.925111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:55.925235 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:55.938089 (+ 12854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:55.938109 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:55.941667 (+ 3558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":242,"threads_started":1}
W20260927 13:56:55.942170 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:55.921133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:48777 (local address 127.22.26.254:44809)
0927 13:56:55.921368 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:55.921374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:55.921597 (+ 223us) server_negotiation.cc:407] Connection header received
0927 13:56:55.921717 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:55.921720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:55.921780 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:55.921859 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:55.922580 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:55.923301 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:55.924492 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:55.924975 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:55.925080 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:55.941915 (+ 16835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:48777: BlockingRecv error: recv got EOF from 127.22.26.193:48777 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 13:56:56 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:56 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:56.267794 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:56.258367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:38583)
0927 13:56:56.258840 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:56.258856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:56.258967 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:56.259293 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:56.259296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:56.259715 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:56.259874 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:56.259883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:56.260864 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:56.260870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:56.262348 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:56.262361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:56.263042 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:56.263047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:56.263244 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:56.263824 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:56.263844 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:56.267508 (+ 3664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":244,"threads_started":1}
W20260927 13:56:56.267963 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:56.258397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:38583 (local address 127.22.26.254:44809)
0927 13:56:56.258612 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:56.258621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:56.258940 (+ 319us) server_negotiation.cc:407] Connection header received
0927 13:56:56.259065 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:56.259069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:56.259148 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:56.259240 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:56.259989 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:56.260714 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:56.262467 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:56.262878 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:56.262973 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:56.267720 (+ 4747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:38583: BlockingRecv error: recv got EOF from 127.22.26.195:38583 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 13:56:56 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:56 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:56.609575 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:56.600524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:50583)
0927 13:56:56.601060 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:56.601076 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:56.601188 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:56.601466 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:56.601470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:56.601880 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:56.602037 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:56.602046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:56.603003 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:56.603007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:56.604041 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:56.604052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:56.604869 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:56.604873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:56.605067 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:56.605743 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:56.605765 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:56.609353 (+ 3588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":291,"threads_started":1}
W20260927 13:56:56.609793 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:56.600567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:50583 (local address 127.22.26.254:44809)
0927 13:56:56.600751 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:56.600755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:56.601148 (+ 393us) server_negotiation.cc:407] Connection header received
0927 13:56:56.601279 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:56.601283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:56.601339 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:56.601418 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:56.602155 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:56.602874 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:56.604181 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:56.604709 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:56.604861 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:56.609514 (+ 4653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:50583: BlockingRecv error: recv got EOF from 127.22.26.194:50583 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 13:56:56 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:56 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:56.952171 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:56.942929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:33743)
0927 13:56:56.943445 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:56.943462 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:56.943612 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:56.943918 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:56.943922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:56.944393 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:56.944598 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:56.944607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:56.946047 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:56.946051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:56.947047 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:56.947058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:56.947750 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:56.947754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:56.947859 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:56.948447 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:56.948463 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:56.951781 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":294,"threads_started":1}
W20260927 13:56:56.952464 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:56.943022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:33743 (local address 127.22.26.254:44809)
0927 13:56:56.943263 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:56.943269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:56.943572 (+ 303us) server_negotiation.cc:407] Connection header received
0927 13:56:56.943721 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:56.943725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:56.943786 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:56.943864 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:56.944747 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:56.945907 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:56.947251 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:56.947625 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:56.947719 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:56.952161 (+ 4442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:33743: BlockingRecv error: recv got EOF from 127.22.26.193:33743 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 13:56:57 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:57 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:57.278469 31926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:57.268805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:41427)
0927 13:56:57.269319 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:57.269356 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:57.269479 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:57.269766 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:57.269770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:57.270166 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:57.270303 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:57.270311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:57.271466 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:57.271470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:57.272451 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:57.272463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:57.273268 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:57.273272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:57.273397 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:57.274122 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:57.274148 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:57.278250 (+ 4102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":305,"threads_started":1}
W20260927 13:56:57.278623 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:57.268976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:41427 (local address 127.22.26.254:44809)
0927 13:56:57.269228 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:57.269232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:57.269422 (+ 190us) server_negotiation.cc:407] Connection header received
0927 13:56:57.269562 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:57.269565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:57.269624 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:57.269713 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:57.270454 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:57.271316 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:57.272593 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:57.273128 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:57.273222 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:57.278410 (+ 5188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:41427: BlockingRecv error: recv got EOF from 127.22.26.195:41427 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 13:56:57 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:57 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:57.619642 31927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:57.610506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:40733)
0927 13:56:57.611063 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:57.611123 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:57.611251 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:57.611561 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:57.611565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:57.611929 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:57.612095 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:57.612104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:57.613332 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:57.613336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:57.614425 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:57.614437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:57.615249 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:57.615253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:57.615367 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:57.616017 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:57.616035 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:57.619358 (+ 3323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":309,"threads_started":1}
W20260927 13:56:57.619796 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:57.610591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:40733 (local address 127.22.26.254:44809)
0927 13:56:57.610779 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:57.610783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:57.611207 (+ 424us) server_negotiation.cc:407] Connection header received
0927 13:56:57.611348 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:57.611351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:57.611410 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:57.611491 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:57.612275 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:57.613147 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:57.614567 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:57.615036 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:57.615235 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:57.619546 (+ 4311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:40733: BlockingRecv error: recv got EOF from 127.22.26.194:40733 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 13:56:57 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:57 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:57.962584 31928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:57.953300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:40061)
0927 13:56:57.954269 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:57.954290 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:57.954434 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:57.954806 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:57.954810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:57.955263 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:57.955438 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:57.955447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:57.956433 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:57.956439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:57.957336 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:57.957347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:57.958061 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:57.958075 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:57.958252 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:57.958804 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:57.958821 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:57.962338 (+ 3517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":703,"threads_started":1}
W20260927 13:56:57.962741 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:57.953513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:40061 (local address 127.22.26.254:44809)
0927 13:56:57.953779 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:57.953785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:57.954377 (+ 592us) server_negotiation.cc:407] Connection header received
0927 13:56:57.954601 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:57.954605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:57.954664 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:57.954768 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:57.955615 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:57.956301 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:57.957495 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:57.957912 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:57.958047 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:57.962506 (+ 4459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:40061: BlockingRecv error: recv got EOF from 127.22.26.193:40061 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Sep 27 13:56:58 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:58 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:58.289335 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:58.279592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:52583)
0927 13:56:58.280221 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:58.280239 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:58.280398 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:58.280797 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:58.280803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:58.281399 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:58.281596 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:58.281607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:58.282683 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:58.282691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:58.283780 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:58.283794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:58.284597 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:58.284604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:58.284793 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:58.285471 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:58.285492 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:58.289068 (+ 3576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":350,"threads_started":1}
W20260927 13:56:58.289492 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:58.279710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:52583 (local address 127.22.26.254:44809)
0927 13:56:58.279917 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:58.279926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:58.280328 (+ 402us) server_negotiation.cc:407] Connection header received
0927 13:56:58.280576 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:58.280580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:58.280645 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:58.280764 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:58.281762 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:58.282533 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:58.283947 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:58.284410 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:58.284583 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:58.289251 (+ 4668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:52583: BlockingRecv error: recv got EOF from 127.22.26.195:52583 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 13:56:58 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:58 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:58.629532 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:58.620600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:45699)
0927 13:56:58.621136 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:58.621153 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:58.621270 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:58.621553 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:58.621556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:58.621966 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:58.622143 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:58.622152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:58.623199 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:58.623203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:58.624128 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:58.624139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:58.624973 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:58.624979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:58.625160 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:58.625818 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:58.625843 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:58.629309 (+ 3466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":318,"threads_started":1}
W20260927 13:56:58.629758 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:58.620666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:45699 (local address 127.22.26.254:44809)
0927 13:56:58.620876 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:58.620881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:58.621227 (+ 346us) server_negotiation.cc:407] Connection header received
0927 13:56:58.621366 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:58.621369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:58.621429 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:58.621511 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:58.622297 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:58.623061 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:58.624273 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:58.624825 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:58.624958 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:58.629534 (+ 4576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:45699: BlockingRecv error: recv got EOF from 127.22.26.194:45699 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 13:56:58 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:58 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:58.972827 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:58.963728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:57259)
0927 13:56:58.964334 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:58.964350 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:58.964464 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:58.964743 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:58.964746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:58.965150 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:58.965318 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:58.965326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:58.966428 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:58.966432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:58.967400 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:58.967411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:58.968226 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:58.968240 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:58.968366 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:58.968928 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:58.968948 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:58.972491 (+ 3543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":312,"threads_started":1}
W20260927 13:56:58.973011 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:58.963809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:57259 (local address 127.22.26.254:44809)
0927 13:56:58.964018 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:58.964028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:58.964412 (+ 384us) server_negotiation.cc:407] Connection header received
0927 13:56:58.964551 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:58.964554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:58.964620 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:58.964701 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:58.965449 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:58.966293 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:58.967540 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:58.968070 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:58.968168 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:58.972758 (+ 4590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:57259: BlockingRecv error: recv got EOF from 127.22.26.193:57259 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 13:56:59 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:59 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:59.299430 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:59.290353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:48323)
0927 13:56:59.290924 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:59.290941 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:59.291068 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:59.291380 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:59.291383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:59.291789 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:59.291971 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:59.291979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:59.293082 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:59.293086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:59.294028 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:59.294040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:59.294775 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:59.294780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:59.294908 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:59.295610 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:59.295627 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:59.299111 (+ 3484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":350,"threads_started":1}
W20260927 13:56:59.299568 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:59.290466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:48323 (local address 127.22.26.254:44809)
0927 13:56:59.290668 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:59.290673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:59.291027 (+ 354us) server_negotiation.cc:407] Connection header received
0927 13:56:59.291177 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:59.291181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:59.291245 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:59.291331 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:59.292129 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:59.292952 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:59.294179 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:59.294634 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:59.294732 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:59.299290 (+ 4558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:48323: BlockingRecv error: recv got EOF from 127.22.26.195:48323 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 13:56:59 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:59 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:59.640931 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:59.630474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:49739)
0927 13:56:59.631131 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:59.631148 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:59.631260 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:59.631572 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:59.631575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:59.631933 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:59.632114 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:59.632122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:59.633263 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:59.633266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:59.634202 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:59.634214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:59.635437 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:59.635448 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:59.635652 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:59.636490 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:59.636513 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:59.640687 (+ 4174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":386,"threads_started":1}
W20260927 13:56:59.641410 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:59.630536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:49739 (local address 127.22.26.254:44809)
0927 13:56:59.630749 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:59.630753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:59.631225 (+ 472us) server_negotiation.cc:407] Connection header received
0927 13:56:59.631363 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:59.631367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:59.631434 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:59.631540 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:59.632268 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:59.633126 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:59.634341 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:59.634871 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:59.635049 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:59.640888 (+ 5839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:49739: BlockingRecv error: recv got EOF from 127.22.26.194:49739 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 13:56:59 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:56:59 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:56:59.983343 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:56:59.973862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:60861)
0927 13:56:59.974490 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 13:56:59.974507 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:56:59.974627 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:56:59.974968 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:56:59.974972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:56:59.975466 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:56:59.975648 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:59.975658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:59.976905 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:59.976910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:59.978115 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:56:59.978131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:56:59.978850 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:56:59.978854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:56:59.978984 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:59.979636 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:56:59.979653 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:56:59.982990 (+ 3337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":400,"threads_started":1}
W20260927 13:56:59.983575 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:56:59.973939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:60861 (local address 127.22.26.254:44809)
0927 13:56:59.974103 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 13:56:59.974107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:56:59.974618 (+ 511us) server_negotiation.cc:407] Connection header received
0927 13:56:59.974733 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:56:59.974737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:56:59.974801 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:56:59.974882 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:56:59.975796 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:59.976738 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:59.978274 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:56:59.978724 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:56:59.978817 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:56:59.983294 (+ 4477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:60861: BlockingRecv error: recv got EOF from 127.22.26.193:60861 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 13:57:00 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:57:00 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:57:00.309676 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:00.300702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:50629)
0927 13:57:00.301315 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:00.301332 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:00.301453 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:00.301760 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:00.301764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:00.302208 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:00.302387 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:00.302397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:00.303554 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:00.303559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:00.304483 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:00.304505 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:00.305235 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:00.305240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:00.305371 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:00.305988 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:57:00.306007 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:00.309431 (+ 3424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":326,"threads_started":1}
W20260927 13:57:00.309908 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:57:00.300749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:50629 (local address 127.22.26.254:44809)
0927 13:57:00.300953 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:00.300959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:00.301413 (+ 454us) server_negotiation.cc:407] Connection header received
0927 13:57:00.301564 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:00.301568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:00.301626 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:00.301705 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:00.302530 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:00.303380 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:00.304629 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:00.305045 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:00.305138 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:00.309638 (+ 4500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:50629: BlockingRecv error: recv got EOF from 127.22.26.195:50629 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 13:57:00 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:57:00 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:57:00.651336 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:00.641904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:40705)
0927 13:57:00.642409 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:00.642424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:00.642541 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:00.642856 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:00.642859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:00.643403 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:00.643636 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:00.643651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:00.644937 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:00.644941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:00.645843 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:00.645854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:00.646620 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:00.646623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:00.646737 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:00.647324 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:57:00.647341 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:00.650962 (+ 3621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":276,"threads_started":1}
W20260927 13:57:00.651448 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:57:00.642073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:40705 (local address 127.22.26.254:44809)
0927 13:57:00.642289 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:00.642293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:00.642508 (+ 215us) server_negotiation.cc:407] Connection header received
0927 13:57:00.642647 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:00.642653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:00.642718 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:00.642837 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:00.643798 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:00.644815 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:00.645990 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:00.646471 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:00.646603 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:00.651204 (+ 4601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:40705: BlockingRecv error: recv got EOF from 127.22.26.194:40705 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 13:57:00 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:57:00 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:57:00.998719 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:00.984575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:49503)
0927 13:57:00.985095 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:00.985111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:00.985220 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:00.985484 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:00.985487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:00.985883 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:00.986064 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:00.986073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:00.987167 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:00.987176 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:00.988382 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:00.988397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:00.989337 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:00.989342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:00.989485 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:00.994814 (+ 5329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:57:00.994833 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:00.998453 (+ 3620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
W20260927 13:57:00.998878 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:57:00.984650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:49503 (local address 127.22.26.254:44809)
0927 13:57:00.984820 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:00.984824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:00.985183 (+ 359us) server_negotiation.cc:407] Connection header received
0927 13:57:00.985303 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:00.985306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:00.985364 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:00.985443 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:00.986187 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:00.986925 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:00.988509 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:00.989201 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:00.989294 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:00.998649 (+ 9355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:49503: BlockingRecv error: recv got EOF from 127.22.26.193:49503 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 13:57:01 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:57:01 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.195@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:57:01.319677 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:01.310634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.195:59249)
0927 13:57:01.311140 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:01.311157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:01.311268 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:01.311604 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:01.311609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:01.312069 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:01.312236 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:01.312246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:01.313371 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:01.313376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:01.314380 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:01.314393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:01.315063 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:01.315068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:01.315252 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:01.315860 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:57:01.315877 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:01.319377 (+ 3500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":245,"threads_started":1}
W20260927 13:57:01.319902 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:57:01.310741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:59249 (local address 127.22.26.254:44809)
0927 13:57:01.311077 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:01.311123 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:01.311229 (+ 106us) server_negotiation.cc:407] Connection header received
0927 13:57:01.311364 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:01.311368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:01.311431 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:01.311513 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:01.312406 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:01.313181 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:01.314517 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:01.314905 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:01.315002 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:01.319641 (+ 4639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.195:59249: BlockingRecv error: recv got EOF from 127.22.26.195:59249 (error 108)
Metrics: {"server-negotiator.queue_time_us":187}
Sep 27 13:57:01 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:57:01 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.194@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:57:01.661778 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:01.652558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.194:56341)
0927 13:57:01.653107 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:01.653124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:01.653246 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:01.653600 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:01.653605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:01.654012 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:01.654265 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:01.654277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:01.655552 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:01.655557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:01.656567 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:01.656580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:01.657433 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:01.657439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:01.657578 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:01.658219 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:57:01.658237 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:01.661535 (+ 3298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":281,"threads_started":1}
W20260927 13:57:01.662005 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:57:01.652726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:56341 (local address 127.22.26.254:44809)
0927 13:57:01.652917 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:01.652922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:01.653272 (+ 350us) server_negotiation.cc:407] Connection header received
0927 13:57:01.653346 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:01.653350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:01.653430 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:01.653535 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:01.654417 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:01.655422 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:01.656690 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:01.657287 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:01.657382 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:01.661737 (+ 4355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.194:56341: BlockingRecv error: recv got EOF from 127.22.26.194:56341 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 13:57:01 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:57:01 dist-test-slave-gkks krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.26.193@KRBTEST.COM for oryx/127.22.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:57:02.008461 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:01.999748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:44809 (local address 127.22.26.193:48487)
0927 13:57:02.000314 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:02.000330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:02.000439 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:02.000742 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:02.000745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:02.001114 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:02.001277 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:02.001285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:02.002215 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:02.002218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:02.003147 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:02.003157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:02.003813 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:02.003817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:02.003933 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:02.004591 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:57:02.004608 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:02.008242 (+ 3634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.26.254:44809: Server oryx/127.22.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":298,"threads_started":1}
W20260927 13:57:02.008689 31897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:57:01.999798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:48487 (local address 127.22.26.254:44809)
0927 13:57:01.999956 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:01.999960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:02.000399 (+ 439us) server_negotiation.cc:407] Connection header received
0927 13:57:02.000529 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:02.000533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:02.000590 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:02.000691 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:02.001391 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:02.002100 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:02.003267 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:02.003692 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:02.003781 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:02.008415 (+ 4634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.26.193:48487: BlockingRecv error: recv got EOF from 127.22.26.193:48487 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:57:02.010053 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 31480
I20260927 13:57:02.023298 31942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:02.011048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.22.26.193:46005)
0927 13:57:02.011621 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:02.011628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:02.011673 (+ 45us) server_negotiation.cc:407] Connection header received
0927 13:57:02.011727 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:02.011733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:02.011956 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:02.012111 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:02.013655 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:02.014869 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:02.016181 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:02.016745 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:02.016883 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:02.019953 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:02.019963 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:02.019978 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:57:02.020016 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:02.022560 (+ 2544us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.26.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 13:57:02.022943 (+ 383us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48168: Request ticket server kudu/127.22.26.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":264,"threads_started":1}
W20260927 13:57:02.023340 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 31480: Not authorized: Client connection negotiation failed: client connection to 127.22.26.193:46005: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.26.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 13:57:02.023415 31942 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48168: Request ticket server kudu/127.22.26.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 13:57:02.023593 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 31480
I20260927 13:57:02.040562 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 31631
I20260927 13:57:02.053324 31943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:02.041602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44336 (local address 127.22.26.194:45231)
0927 13:57:02.042224 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:02.042233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:02.042266 (+ 33us) server_negotiation.cc:407] Connection header received
0927 13:57:02.042335 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:02.042344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:02.042650 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:02.042861 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:02.044283 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:02.045670 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:02.046835 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:02.047509 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:02.047608 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:02.050142 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:02.050151 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:02.050163 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:57:02.050203 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:02.052842 (+ 2639us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.26.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 13:57:02.053103 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44336: Request ticket server kudu/127.22.26.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":263,"threads_started":1}
W20260927 13:57:02.053445 31943 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44336: Request ticket server kudu/127.22.26.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 13:57:02.053458 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 31631: Not authorized: Client connection negotiation failed: client connection to 127.22.26.194:45231: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.26.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 13:57:02.053610 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 31631
I20260927 13:57:02.070055 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 31764
I20260927 13:57:02.082096 31944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:02.071132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58346 (local address 127.22.26.195:34337)
0927 13:57:02.071655 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:02.071662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:02.071685 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:57:02.071744 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:02.071750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:02.071979 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:02.072195 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:02.073416 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:02.074384 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:02.075716 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:02.076259 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:02.076410 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:02.078951 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:02.078959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:02.078971 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:57:02.079022 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:02.081420 (+ 2398us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.26.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 13:57:02.081772 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58346: Request ticket server kudu/127.22.26.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":239,"threads_started":1}
W20260927 13:57:02.082091 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 31764: Not authorized: Client connection negotiation failed: client connection to 127.22.26.195:34337: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.26.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 13:57:02.082207 31944 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58346: Request ticket server kudu/127.22.26.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 13:57:02.082242 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 31764
I20260927 13:57:02.098508 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 31411
I20260927 13:57:02.099682 31473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:57:02.217402 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 31411
2026-09-27T13:57:02Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21507 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T13:57:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:57:02Z Disabled control of system clock
I20260927 13:57:02.293026 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:40585
--webserver_interface=127.22.26.254
--webserver_port=0
--builtin_ntp_servers=127.22.26.212:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:40585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 13:57:02.614565 31949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:57:02.614842 31949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:57:02.614920 31949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:57:02.622171 31949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:57:02.622278 31949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:57:02.622318 31949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:57:02.622366 31949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:57:02.622411 31949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:57:02.631390 31949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:40585
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:40585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:57:02.632638 31949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:57:02.634348 31949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:57:02.646796 31954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:02.646832 31957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:02.647423 31949 server_base.cc:1034] running on GCE node
W20260927 13:57:02.647478 31955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:02.648340 31949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:57:02.649699 31949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:57:02.650931 31949 hybrid_clock.cc:648] HybridClock initialized: now 1790517422650888 us; error 64 us; skew 500 ppm
I20260927 13:57:02.651355 31949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:57:02.653714 31949 webserver.cc:460] Webserver started at http://127.22.26.254:45101/ using document root <none> and password file <none>
I20260927 13:57:02.654454 31949 fs_manager.cc:359] Metadata directory not provided
I20260927 13:57:02.654594 31949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:57:02.654927 31949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:57:02.657903 31949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "68f8625a49f84e909ea41b54af55ba86"
format_stamp: "Formatted at 2026-09-27 13:57:02 on dist-test-slave-gkks"
server_key: "99b770ae6e30414f84ab3526937d1bb3"
server_key_iv: "f01a20413dac4832b9da4a33da417a6a"
server_key_version: "encryptionkey@0"
I20260927 13:57:02.658715 31949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "68f8625a49f84e909ea41b54af55ba86"
format_stamp: "Formatted at 2026-09-27 13:57:02 on dist-test-slave-gkks"
server_key: "99b770ae6e30414f84ab3526937d1bb3"
server_key_iv: "f01a20413dac4832b9da4a33da417a6a"
server_key_version: "encryptionkey@0"
I20260927 13:57:02.664115 31949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:57:02.667626 31963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:02.668939 31949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 13:57:02.669128 31949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "68f8625a49f84e909ea41b54af55ba86"
format_stamp: "Formatted at 2026-09-27 13:57:02 on dist-test-slave-gkks"
server_key: "99b770ae6e30414f84ab3526937d1bb3"
server_key_iv: "f01a20413dac4832b9da4a33da417a6a"
server_key_version: "encryptionkey@0"
I20260927 13:57:02.669353 31949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:57:02.687997 31949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:57:02.688859 31949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:57:02.689177 31949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:57:02.708115 31949 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:40585
I20260927 13:57:02.708151 32014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:40585 every 8 connection(s)
I20260927 13:57:02.709533 31949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:57:02.712250 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 31949
I20260927 13:57:02.712466 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:57:02.713300 22635 external_mini_cluster.cc:1428] Setting key b39d5a84441a6b65ae811f0cb9573199
I20260927 13:57:02.714123 32015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:57:02.726490 32015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86: Bootstrap starting.
I20260927 13:57:02.727418 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:02.715539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49656 (local address 127.22.26.254:40585)
0927 13:57:02.716500 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:02.716511 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:02.716553 (+ 42us) server_negotiation.cc:407] Connection header received
0927 13:57:02.718702 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:02.718723 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:02.719264 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:02.719691 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:02.720293 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:02.721546 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:02.722884 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:02.723621 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:02.723849 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:02.725062 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:02.725069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:02.725082 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:02.725604 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:02.725689 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:02.725715 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:02.725842 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:02.726081 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 13:57:02.726374 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":337,"threads_started":1}
I20260927 13:57:02.729657 32015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86: Neither blocks nor log segments found. Creating new log.
I20260927 13:57:02.730775 32015 log.cc:824] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86: Log is configured to *not* fsync() on all Append() calls
I20260927 13:57:02.733650 32015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86: No bootstrap required, opened a new log
I20260927 13:57:02.741045 32015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f8625a49f84e909ea41b54af55ba86" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 40585 } }
I20260927 13:57:02.741425 32015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:57:02.741530 32015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f8625a49f84e909ea41b54af55ba86, State: Initialized, Role: FOLLOWER
I20260927 13:57:02.742143 32015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f8625a49f84e909ea41b54af55ba86" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 40585 } }
I20260927 13:57:02.742385 32015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:57:02.742524 32015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:57:02.742710 32015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:57:02.744976 32015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f8625a49f84e909ea41b54af55ba86" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 40585 } }
I20260927 13:57:02.745591 32015 leader_election.cc:302] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68f8625a49f84e909ea41b54af55ba86; no voters:
I20260927 13:57:02.746227 32015 leader_election.cc:288] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:57:02.746356 32020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:57:02.747066 32020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [term 1 LEADER]: Becoming Leader. State: Replica: 68f8625a49f84e909ea41b54af55ba86, State: Running, Role: LEADER
I20260927 13:57:02.748113 32020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f8625a49f84e909ea41b54af55ba86" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 40585 } }
I20260927 13:57:02.748549 32015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:57:02.750850 32021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68f8625a49f84e909ea41b54af55ba86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f8625a49f84e909ea41b54af55ba86" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 40585 } } }
I20260927 13:57:02.751291 32021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [sys.catalog]: This master's current role is: LEADER
I20260927 13:57:02.751147 32022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f8625a49f84e909ea41b54af55ba86. Latest consensus state: current_term: 1 leader_uuid: "68f8625a49f84e909ea41b54af55ba86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f8625a49f84e909ea41b54af55ba86" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 40585 } } }
I20260927 13:57:02.751451 32022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86 [sys.catalog]: This master's current role is: LEADER
I20260927 13:57:02.753425 32029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:57:02.758641 32029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:57:02.766726 32029 catalog_manager.cc:1347] Generated new cluster ID: 0de729c1948142ed8c4f8425f691da12
I20260927 13:57:02.766839 32029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:57:02.780537 32029 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:57:02.782281 32029 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:57:02.789335 32029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68f8625a49f84e909ea41b54af55ba86: Generated new TSK 0
I20260927 13:57:02.790292 32029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:57:02.797564 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:40585
--builtin_ntp_servers=127.22.26.212:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 13:57:03.128811 32039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:57:03.129117 32039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:57:03.129204 32039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:57:03.136507 32039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:57:03.136619 32039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:57:03.136725 32039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:57:03.145649 32039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:40585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:57:03.146997 32039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:57:03.148965 32039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:57:03.162401 32044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:03.162396 32045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:03.162606 32047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:03.163287 32039 server_base.cc:1034] running on GCE node
I20260927 13:57:03.163851 32039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:57:03.164746 32039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:57:03.165973 32039 hybrid_clock.cc:648] HybridClock initialized: now 1790517423165935 us; error 53 us; skew 500 ppm
I20260927 13:57:03.166290 32039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:57:03.168581 32039 webserver.cc:460] Webserver started at http://127.22.26.193:34731/ using document root <none> and password file <none>
I20260927 13:57:03.169274 32039 fs_manager.cc:359] Metadata directory not provided
I20260927 13:57:03.169399 32039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:57:03.169730 32039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:57:03.172554 32039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "c6c02002ce954e57b791a88e95ddaa8a"
format_stamp: "Formatted at 2026-09-27 13:57:03 on dist-test-slave-gkks"
server_key: "5a8bafe09496e8e47966337e895235e3"
server_key_iv: "77e4a7a61d488b6b5006788f981e6d52"
server_key_version: "encryptionkey@0"
I20260927 13:57:03.173395 32039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "c6c02002ce954e57b791a88e95ddaa8a"
format_stamp: "Formatted at 2026-09-27 13:57:03 on dist-test-slave-gkks"
server_key: "5a8bafe09496e8e47966337e895235e3"
server_key_iv: "77e4a7a61d488b6b5006788f981e6d52"
server_key_version: "encryptionkey@0"
I20260927 13:57:03.178783 32039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:57:03.182441 32053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:03.183907 32039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:57:03.184093 32039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "c6c02002ce954e57b791a88e95ddaa8a"
format_stamp: "Formatted at 2026-09-27 13:57:03 on dist-test-slave-gkks"
server_key: "5a8bafe09496e8e47966337e895235e3"
server_key_iv: "77e4a7a61d488b6b5006788f981e6d52"
server_key_version: "encryptionkey@0"
I20260927 13:57:03.184336 32039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:57:03.211439 32039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:57:03.212280 32039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:57:03.212579 32039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:57:03.213549 32039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:57:03.214989 32039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:57:03.215126 32039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:03.215257 32039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:57:03.215346 32039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:03.248335 32039 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:43763
I20260927 13:57:03.248448 32165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:43763 every 8 connection(s)
I20260927 13:57:03.249776 32039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:57:03.260178 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 32039
I20260927 13:57:03.260406 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:57:03.261075 22635 external_mini_cluster.cc:1428] Setting key 70a185cabebcc2ce534c1954a3781fc9
I20260927 13:57:03.262764 32168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:03.252481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:57513 (local address 127.22.26.254:40585)
0927 13:57:03.252933 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:03.252938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:03.254101 (+ 1163us) server_negotiation.cc:407] Connection header received
0927 13:57:03.255928 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:03.255933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:03.256018 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:03.256154 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:03.257402 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:03.258172 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:03.259859 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:03.260342 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:03.260488 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:03.261891 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:03.261897 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:03.261901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:03.261927 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:03.261971 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:03.261983 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:03.262084 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:03.262304 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 13:57:03.262400 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":242,"threads_started":1}
I20260927 13:57:03.263798 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:03.253094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:40585 (local address 127.22.26.193:57513)
0927 13:57:03.253935 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:03.253977 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:03.255666 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:03.256367 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:03.256375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:03.256463 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:03.257001 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:03.257016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:03.258336 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:03.258342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:03.259666 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:03.259679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:03.260581 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:03.260591 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:03.260837 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:03.261572 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:57:03.261574 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:03.261613 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:57:03.261691 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:57:03.262084 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:57:03.262097 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:57:03.262107 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 13:57:03.262297 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 13:57:03.262673 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":241,"threads_started":1}
I20260927 13:57:03.263908 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:40585
--builtin_ntp_servers=127.22.26.212:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 13:57:03.265386 32166 heartbeater.cc:347] Connected to a master server at 127.22.26.254:40585
I20260927 13:57:03.265695 32166 heartbeater.cc:464] Registering TS with master...
I20260927 13:57:03.266396 32166 heartbeater.cc:511] Master 127.22.26.254:40585 requested a full tablet report, sending...
I20260927 13:57:03.268525 31980 ts_manager.cc:194] Registered new tserver with Master: c6c02002ce954e57b791a88e95ddaa8a (127.22.26.193:43763)
I20260927 13:57:03.270679 31980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.26.193:57513
W20260927 13:57:03.594084 32170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:57:03.594348 32170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:57:03.594431 32170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:57:03.601639 32170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:57:03.601727 32170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:57:03.601824 32170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:57:03.610811 32170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:40585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:57:03.612143 32170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:57:03.613943 32170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:57:03.626853 32176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:03.626893 32178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:03.627439 32175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:03.627902 32170 server_base.cc:1034] running on GCE node
I20260927 13:57:03.628469 32170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:57:03.629400 32170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:57:03.630621 32170 hybrid_clock.cc:648] HybridClock initialized: now 1790517423630577 us; error 58 us; skew 500 ppm
I20260927 13:57:03.630931 32170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:57:03.633184 32170 webserver.cc:460] Webserver started at http://127.22.26.194:40185/ using document root <none> and password file <none>
I20260927 13:57:03.633839 32170 fs_manager.cc:359] Metadata directory not provided
I20260927 13:57:03.633967 32170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:57:03.634287 32170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:57:03.637060 32170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "cac7b7f5f52e40daa59271eed8019900"
format_stamp: "Formatted at 2026-09-27 13:57:03 on dist-test-slave-gkks"
server_key: "a46598e3c7f1eb31a1ae770c4ba82bbc"
server_key_iv: "950c09c3e8d53c25efb9c9da0db6dec1"
server_key_version: "encryptionkey@0"
I20260927 13:57:03.637825 32170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "cac7b7f5f52e40daa59271eed8019900"
format_stamp: "Formatted at 2026-09-27 13:57:03 on dist-test-slave-gkks"
server_key: "a46598e3c7f1eb31a1ae770c4ba82bbc"
server_key_iv: "950c09c3e8d53c25efb9c9da0db6dec1"
server_key_version: "encryptionkey@0"
I20260927 13:57:03.642639 32170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 13:57:03.645956 32184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:03.647235 32170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:57:03.647399 32170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "cac7b7f5f52e40daa59271eed8019900"
format_stamp: "Formatted at 2026-09-27 13:57:03 on dist-test-slave-gkks"
server_key: "a46598e3c7f1eb31a1ae770c4ba82bbc"
server_key_iv: "950c09c3e8d53c25efb9c9da0db6dec1"
server_key_version: "encryptionkey@0"
I20260927 13:57:03.647629 32170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:57:03.670130 32170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:57:03.670985 32170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:57:03.671324 32170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:57:03.672276 32170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:57:03.673748 32170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:57:03.673871 32170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:03.674007 32170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:57:03.674096 32170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:03.707875 32170 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:40303
I20260927 13:57:03.707996 32296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:40303 every 8 connection(s)
I20260927 13:57:03.709421 32170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:57:03.718456 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 32170
I20260927 13:57:03.718659 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:57:03.719226 22635 external_mini_cluster.cc:1428] Setting key 8e4fb2c9eddbc11b8b845d2661820196
I20260927 13:57:03.722280 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:40585
--builtin_ntp_servers=127.22.26.212:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 13:57:03.723309 32168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:03.712128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:43869 (local address 127.22.26.254:40585)
0927 13:57:03.712337 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:03.712343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:03.713883 (+ 1540us) server_negotiation.cc:407] Connection header received
0927 13:57:03.715629 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:03.715634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:03.715714 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:03.715829 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:03.716820 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:03.717936 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:03.720220 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:03.720874 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:03.721007 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:03.722464 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:03.722473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:03.722480 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:03.722506 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:03.722558 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:03.722573 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:03.722657 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:03.722876 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 13:57:03.722970 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:57:03.724359 32299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:03.712657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:40585 (local address 127.22.26.194:43869)
0927 13:57:03.713704 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:03.713747 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:03.715375 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:03.716009 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:03.716016 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:03.716098 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:03.716671 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:03.716687 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:03.718125 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:03.718134 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:03.720068 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:03.720084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:03.721186 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:03.721194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:03.721419 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:03.722147 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:57:03.722150 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:03.722187 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:57:03.722264 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:57:03.722658 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:57:03.722671 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:57:03.722680 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 13:57:03.722860 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 13:57:03.723279 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":330,"threads_started":1}
I20260927 13:57:03.725732 32297 heartbeater.cc:347] Connected to a master server at 127.22.26.254:40585
I20260927 13:57:03.726037 32297 heartbeater.cc:464] Registering TS with master...
I20260927 13:57:03.726752 32297 heartbeater.cc:511] Master 127.22.26.254:40585 requested a full tablet report, sending...
I20260927 13:57:03.728117 31980 ts_manager.cc:194] Registered new tserver with Master: cac7b7f5f52e40daa59271eed8019900 (127.22.26.194:40303)
I20260927 13:57:03.728925 31980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.26.194:43869
W20260927 13:57:04.054894 32300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:57:04.055215 32300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:57:04.055305 32300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:57:04.062237 32300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:57:04.062322 32300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:57:04.062424 32300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:57:04.071210 32300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:40585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:57:04.072368 32300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:57:04.074219 32300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:57:04.087060 32305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:04.087128 32306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:04.087128 32308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:04.088004 32300 server_base.cc:1034] running on GCE node
I20260927 13:57:04.088646 32300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:57:04.089669 32300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:57:04.090906 32300 hybrid_clock.cc:648] HybridClock initialized: now 1790517424090870 us; error 48 us; skew 500 ppm
I20260927 13:57:04.091300 32300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:57:04.093510 32300 webserver.cc:460] Webserver started at http://127.22.26.195:40533/ using document root <none> and password file <none>
I20260927 13:57:04.094228 32300 fs_manager.cc:359] Metadata directory not provided
I20260927 13:57:04.094383 32300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:57:04.094723 32300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:57:04.097584 32300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "0e5aa783254d43febd5815860666cd77"
format_stamp: "Formatted at 2026-09-27 13:57:04 on dist-test-slave-gkks"
server_key: "2ead03e637de9a9c52fc2913dc72f85f"
server_key_iv: "a9d22b5f82964a2e1dfc465a8b95bb43"
server_key_version: "encryptionkey@0"
I20260927 13:57:04.098356 32300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "0e5aa783254d43febd5815860666cd77"
format_stamp: "Formatted at 2026-09-27 13:57:04 on dist-test-slave-gkks"
server_key: "2ead03e637de9a9c52fc2913dc72f85f"
server_key_iv: "a9d22b5f82964a2e1dfc465a8b95bb43"
server_key_version: "encryptionkey@0"
I20260927 13:57:04.103273 32300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 13:57:04.106547 32314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:04.107945 32300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:57:04.108136 32300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "0e5aa783254d43febd5815860666cd77"
format_stamp: "Formatted at 2026-09-27 13:57:04 on dist-test-slave-gkks"
server_key: "2ead03e637de9a9c52fc2913dc72f85f"
server_key_iv: "a9d22b5f82964a2e1dfc465a8b95bb43"
server_key_version: "encryptionkey@0"
I20260927 13:57:04.108364 32300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:57:04.139608 32300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:57:04.140517 32300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:57:04.140831 32300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:57:04.141819 32300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:57:04.143347 32300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:57:04.143492 32300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:04.143620 32300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:57:04.143705 32300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:04.177584 32300 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:37357
I20260927 13:57:04.177649 32426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:37357 every 8 connection(s)
I20260927 13:57:04.178887 32300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:57:04.186645 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 32300
I20260927 13:57:04.186838 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:57:04.187377 22635 external_mini_cluster.cc:1428] Setting key 048729cc1df4b0b678d60339f658d275
I20260927 13:57:04.192000 32168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:04.181483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:48777 (local address 127.22.26.254:40585)
0927 13:57:04.181652 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:04.181656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:04.182958 (+ 1302us) server_negotiation.cc:407] Connection header received
0927 13:57:04.184785 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:04.184790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:04.184859 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:04.184948 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:04.185957 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:04.186690 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:04.188976 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:04.189617 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:04.189761 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:04.191173 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:04.191179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:04.191184 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:04.191211 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:04.191242 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:04.191255 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:04.191346 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:04.191572 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 13:57:04.191678 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:57:04.193019 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:04.181967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:40585 (local address 127.22.26.195:48777)
0927 13:57:04.182834 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:04.182870 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:04.184496 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:04.185172 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:04.185179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:04.185264 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:04.185816 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:04.185832 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:04.186878 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:04.186885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:04.188819 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:04.188839 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:04.189843 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:04.189851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:04.190096 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:04.190831 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:57:04.190834 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:04.190873 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:57:04.190948 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:57:04.191363 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:57:04.191373 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:57:04.191381 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 13:57:04.191562 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 13:57:04.191992 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":260,"threads_started":1}
I20260927 13:57:04.194455 32427 heartbeater.cc:347] Connected to a master server at 127.22.26.254:40585
I20260927 13:57:04.194741 32427 heartbeater.cc:464] Registering TS with master...
I20260927 13:57:04.195571 32427 heartbeater.cc:511] Master 127.22.26.254:40585 requested a full tablet report, sending...
I20260927 13:57:04.196962 31980 ts_manager.cc:194] Registered new tserver with Master: 0e5aa783254d43febd5815860666cd77 (127.22.26.195:37357)
I20260927 13:57:04.197745 31980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.26.195:48777
I20260927 13:57:04.202595 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 13:57:04.208823 32168 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:57:04.206962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.22.26.254:40585)
0927 13:57:04.207231 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:04.207240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:04.207742 (+ 502us) server_negotiation.cc:407] Connection header received
0927 13:57:04.207875 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:04.207878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:04.207954 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:04.208045 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:04.208567 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49668: BlockingRecv error: recv got EOF from 127.0.0.1:49668 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 13:57:04.209509 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 32039
I20260927 13:57:04.217262 32436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:04.210384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.22.26.193:43763)
0927 13:57:04.210926 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:04.210933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:04.210958 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:57:04.211030 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:04.211036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:04.211306 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:04.211425 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:04.211906 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:04.213086 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:04.214250 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:04.214764 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:04.214860 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:04.215895 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:04.215903 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:04.215914 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:04.216479 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:04.216556 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:04.216575 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:04.216676 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:04.216842 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 13:57:04.216955 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":295,"threads_started":1}
I20260927 13:57:04.218566 32161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:57:04.301496 32166 heartbeater.cc:503] Master 127.22.26.254:40585 was elected leader, sending a full tablet report...
I20260927 13:57:04.303798 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 32039
I20260927 13:57:04.320554 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 32170
I20260927 13:57:04.328372 32438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:04.321472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.22.26.194:40303)
0927 13:57:04.322014 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:04.322029 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:04.322058 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:57:04.322126 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:04.322133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:04.322358 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:04.322495 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:04.322858 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:04.323912 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:04.325133 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:04.325699 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:04.325797 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:04.326762 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:04.326770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:04.326783 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:04.327624 (+ 841us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:04.327686 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:04.327706 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:04.327793 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:04.327934 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 13:57:04.328060 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":292,"threads_started":1}
I20260927 13:57:04.329658 32292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:57:04.412130 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 32170
I20260927 13:57:04.429008 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 32300
I20260927 13:57:04.437402 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:04.430163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.22.26.195:37357)
0927 13:57:04.430819 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:04.430826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:04.430857 (+ 31us) server_negotiation.cc:407] Connection header received
0927 13:57:04.430917 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:04.430923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:04.431235 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:04.431365 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:04.431797 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:04.432837 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:04.434161 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:04.434740 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:04.434835 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:04.435951 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:04.435960 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:04.435973 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:04.436581 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:04.436627 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:04.436647 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:04.436750 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:04.436885 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 13:57:04.437023 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":330,"threads_started":1}
I20260927 13:57:04.438766 32422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:57:04.523232 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 32300
I20260927 13:57:04.539589 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 31949
I20260927 13:57:04.540865 32010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:57:04.645903 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 31949
2026-09-27T13:57:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2423 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T13:57:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:57:04Z Disabled control of system clock
I20260927 13:57:04.715050 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42363
--webserver_interface=127.22.26.254
--webserver_port=0
--builtin_ntp_servers=127.22.26.212:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:42363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 13:57:05.034552 32446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:57:05.034821 32446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:57:05.034905 32446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:57:05.041754 32446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:57:05.041843 32446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:57:05.041882 32446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:57:05.041929 32446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:57:05.041972 32446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:57:05.050787 32446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:42363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:57:05.052040 32446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:57:05.053838 32446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:57:05.065362 32451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:05.065411 32452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:05.065462 32454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:05.068529 32446 server_base.cc:1034] running on GCE node
I20260927 13:57:05.069269 32446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:57:05.070602 32446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:57:05.071838 32446 hybrid_clock.cc:648] HybridClock initialized: now 1790517425071806 us; error 48 us; skew 500 ppm
I20260927 13:57:05.072186 32446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:57:05.074338 32446 webserver.cc:460] Webserver started at http://127.22.26.254:41581/ using document root <none> and password file <none>
I20260927 13:57:05.075016 32446 fs_manager.cc:359] Metadata directory not provided
I20260927 13:57:05.075170 32446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:57:05.075492 32446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:57:05.078259 32446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "ef06a324ae8f4cc3919822d1ac30107a"
format_stamp: "Formatted at 2026-09-27 13:57:05 on dist-test-slave-gkks"
server_key: "355984f1639126242230d73d800f2125"
server_key_iv: "fe70cdd9ea87dfdd844af98289571c07"
server_key_version: "encryptionkey@0"
I20260927 13:57:05.079025 32446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "ef06a324ae8f4cc3919822d1ac30107a"
format_stamp: "Formatted at 2026-09-27 13:57:05 on dist-test-slave-gkks"
server_key: "355984f1639126242230d73d800f2125"
server_key_iv: "fe70cdd9ea87dfdd844af98289571c07"
server_key_version: "encryptionkey@0"
I20260927 13:57:05.083988 32446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 13:57:05.087327 32460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:05.088654 32446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:57:05.088831 32446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "ef06a324ae8f4cc3919822d1ac30107a"
format_stamp: "Formatted at 2026-09-27 13:57:05 on dist-test-slave-gkks"
server_key: "355984f1639126242230d73d800f2125"
server_key_iv: "fe70cdd9ea87dfdd844af98289571c07"
server_key_version: "encryptionkey@0"
I20260927 13:57:05.089053 32446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:57:05.094751 32446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:57:05.095502 32446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:57:05.095811 32446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:57:05.113667 32446 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:42363
I20260927 13:57:05.113714 32511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:42363 every 8 connection(s)
I20260927 13:57:05.114964 32446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:57:05.118463 32512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:57:05.123800 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 32446
I20260927 13:57:05.123978 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:57:05.124784 22635 external_mini_cluster.cc:1428] Setting key 1f73aedb49bb0c0e081afd17aa250b0f
I20260927 13:57:05.129515 32512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a: Bootstrap starting.
I20260927 13:57:05.133116 32512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a: Neither blocks nor log segments found. Creating new log.
I20260927 13:57:05.133194 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:05.126846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.22.26.254:42363)
0927 13:57:05.127939 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:05.127950 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:05.127989 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:57:05.129570 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:05.129589 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:05.130010 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:05.130314 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:05.131414 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:05.131420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:05.131431 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:05.131448 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:05.131501 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:05.131517 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:05.131735 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:05.131863 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 13:57:05.132124 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":469,"threads_started":1}
I20260927 13:57:05.134465 32512 log.cc:824] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a: Log is configured to *not* fsync() on all Append() calls
I20260927 13:57:05.137236 32512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a: No bootstrap required, opened a new log
I20260927 13:57:05.144177 32512 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef06a324ae8f4cc3919822d1ac30107a" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42363 } }
I20260927 13:57:05.144521 32512 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:57:05.144629 32512 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef06a324ae8f4cc3919822d1ac30107a, State: Initialized, Role: FOLLOWER
I20260927 13:57:05.145133 32512 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef06a324ae8f4cc3919822d1ac30107a" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42363 } }
I20260927 13:57:05.145372 32512 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:57:05.145507 32512 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:57:05.145673 32512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:57:05.147821 32512 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef06a324ae8f4cc3919822d1ac30107a" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42363 } }
I20260927 13:57:05.148408 32512 leader_election.cc:302] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef06a324ae8f4cc3919822d1ac30107a; no voters:
I20260927 13:57:05.148952 32512 leader_election.cc:288] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:57:05.149078 32517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:57:05.149837 32517 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [term 1 LEADER]: Becoming Leader. State: Replica: ef06a324ae8f4cc3919822d1ac30107a, State: Running, Role: LEADER
I20260927 13:57:05.150420 32517 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef06a324ae8f4cc3919822d1ac30107a" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42363 } }
I20260927 13:57:05.150909 32512 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:57:05.153153 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef06a324ae8f4cc3919822d1ac30107a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef06a324ae8f4cc3919822d1ac30107a" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42363 } } }
I20260927 13:57:05.153587 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [sys.catalog]: This master's current role is: LEADER
I20260927 13:57:05.153365 32519 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef06a324ae8f4cc3919822d1ac30107a. Latest consensus state: current_term: 1 leader_uuid: "ef06a324ae8f4cc3919822d1ac30107a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef06a324ae8f4cc3919822d1ac30107a" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42363 } } }
I20260927 13:57:05.153759 32519 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a [sys.catalog]: This master's current role is: LEADER
I20260927 13:57:05.154567 32525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:57:05.160538 32525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:57:05.168083 32525 catalog_manager.cc:1347] Generated new cluster ID: 8771e2527fd841ef8a7367d295c446e7
I20260927 13:57:05.168186 32525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:57:05.197413 32525 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:57:05.197854 32525 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:57:05.207367 32525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef06a324ae8f4cc3919822d1ac30107a: Generated new TSK 0
I20260927 13:57:05.208163 32525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:57:05.221225 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:42363
--builtin_ntp_servers=127.22.26.212:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 13:57:05.547565 32536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:57:05.547814 32536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:57:05.547899 32536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:57:05.555256 32536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:57:05.555339 32536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:57:05.555462 32536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:57:05.564193 32536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:42363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:57:05.565366 32536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:57:05.567003 32536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:57:05.579602 32541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:05.579612 32542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:05.579675 32544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:05.580502 32536 server_base.cc:1034] running on GCE node
I20260927 13:57:05.581153 32536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:57:05.582151 32536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:57:05.583375 32536 hybrid_clock.cc:648] HybridClock initialized: now 1790517425583337 us; error 34 us; skew 500 ppm
I20260927 13:57:05.583698 32536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:57:05.586066 32536 webserver.cc:460] Webserver started at http://127.22.26.193:37267/ using document root <none> and password file <none>
I20260927 13:57:05.586760 32536 fs_manager.cc:359] Metadata directory not provided
I20260927 13:57:05.586897 32536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:57:05.587291 32536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:57:05.590093 32536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "be5bd95d80a54ab2b91ff0c7078b89a6"
format_stamp: "Formatted at 2026-09-27 13:57:05 on dist-test-slave-gkks"
server_key: "96b96d9a57a25464ee84f02bcd07bfb8"
server_key_iv: "9763b24a4aec71ca3b08fdcc6f6e40c6"
server_key_version: "encryptionkey@0"
I20260927 13:57:05.590888 32536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "be5bd95d80a54ab2b91ff0c7078b89a6"
format_stamp: "Formatted at 2026-09-27 13:57:05 on dist-test-slave-gkks"
server_key: "96b96d9a57a25464ee84f02bcd07bfb8"
server_key_iv: "9763b24a4aec71ca3b08fdcc6f6e40c6"
server_key_version: "encryptionkey@0"
I20260927 13:57:05.596231 32536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 13:57:05.600032 32550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:05.601562 32536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:57:05.601802 32536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "be5bd95d80a54ab2b91ff0c7078b89a6"
format_stamp: "Formatted at 2026-09-27 13:57:05 on dist-test-slave-gkks"
server_key: "96b96d9a57a25464ee84f02bcd07bfb8"
server_key_iv: "9763b24a4aec71ca3b08fdcc6f6e40c6"
server_key_version: "encryptionkey@0"
I20260927 13:57:05.602046 32536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:57:05.607854 32536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:57:05.608678 32536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:57:05.608951 32536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:57:05.609872 32536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:57:05.611325 32536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:57:05.611449 32536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:05.611570 32536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:57:05.611667 32536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:05.644126 32536 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:45959
I20260927 13:57:05.644178 32662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:45959 every 8 connection(s)
I20260927 13:57:05.645454 32536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:57:05.653815 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:05.648115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:41533 (local address 127.22.26.254:42363)
0927 13:57:05.648621 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:05.648626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:05.649683 (+ 1057us) server_negotiation.cc:407] Connection header received
0927 13:57:05.651481 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:05.651487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:05.651566 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:05.651712 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:05.653057 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:05.653063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:05.653067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:05.653092 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:05.653132 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:05.653146 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:05.653274 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:05.653461 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 13:57:05.653533 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":297,"threads_started":1}
I20260927 13:57:05.654749 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:05.648656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42363 (local address 127.22.26.193:41533)
0927 13:57:05.649525 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:05.649560 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:05.651205 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:05.651936 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:05.651944 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:05.652064 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:05.652767 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:57:05.652770 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:05.652807 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:57:05.652887 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:57:05.653241 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:57:05.653253 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:57:05.653261 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 13:57:05.653535 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 13:57:05.653770 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":257,"threads_started":1}
I20260927 13:57:05.655867 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 32536
I20260927 13:57:05.656097 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:57:05.656548 32663 heartbeater.cc:347] Connected to a master server at 127.22.26.254:42363
I20260927 13:57:05.656772 22635 external_mini_cluster.cc:1428] Setting key bc9347b07d887e4ec4aeda01e72d9592
I20260927 13:57:05.656867 32663 heartbeater.cc:464] Registering TS with master...
I20260927 13:57:05.657449 32663 heartbeater.cc:511] Master 127.22.26.254:42363 requested a full tablet report, sending...
I20260927 13:57:05.659474 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:42363
--builtin_ntp_servers=127.22.26.212:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 13:57:05.659489 32477 ts_manager.cc:194] Registered new tserver with Master: be5bd95d80a54ab2b91ff0c7078b89a6 (127.22.26.193:45959)
W20260927 13:57:05.977115 32667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:57:05.977362 32667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:57:05.977540 32667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:57:05.984558 32667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:57:05.984645 32667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:57:05.984814 32667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:57:05.993770 32667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:42363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:57:05.994987 32667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:57:05.996783 32667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:57:06.009773 32672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:06.009753 32675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:06.010092 32673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:06.010592 32667 server_base.cc:1034] running on GCE node
I20260927 13:57:06.011247 32667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:57:06.012115 32667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:57:06.013319 32667 hybrid_clock.cc:648] HybridClock initialized: now 1790517426013277 us; error 60 us; skew 500 ppm
I20260927 13:57:06.013651 32667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:57:06.020848 32667 webserver.cc:460] Webserver started at http://127.22.26.194:39503/ using document root <none> and password file <none>
I20260927 13:57:06.021561 32667 fs_manager.cc:359] Metadata directory not provided
I20260927 13:57:06.021682 32667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:57:06.022022 32667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:57:06.024919 32667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "d8a89c53a82e441fad9b349bb853665b"
format_stamp: "Formatted at 2026-09-27 13:57:06 on dist-test-slave-gkks"
server_key: "f33669b4b012bd7b73d6a2263246cc40"
server_key_iv: "434a06148d9925f786bcb9e9c3c00472"
server_key_version: "encryptionkey@0"
I20260927 13:57:06.025707 32667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "d8a89c53a82e441fad9b349bb853665b"
format_stamp: "Formatted at 2026-09-27 13:57:06 on dist-test-slave-gkks"
server_key: "f33669b4b012bd7b73d6a2263246cc40"
server_key_iv: "434a06148d9925f786bcb9e9c3c00472"
server_key_version: "encryptionkey@0"
I20260927 13:57:06.031148 32667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 13:57:06.034317 32681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:06.035678 32667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 13:57:06.035845 32667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "d8a89c53a82e441fad9b349bb853665b"
format_stamp: "Formatted at 2026-09-27 13:57:06 on dist-test-slave-gkks"
server_key: "f33669b4b012bd7b73d6a2263246cc40"
server_key_iv: "434a06148d9925f786bcb9e9c3c00472"
server_key_version: "encryptionkey@0"
I20260927 13:57:06.036059 32667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:57:06.041636 32667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:57:06.042389 32667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:57:06.042658 32667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:57:06.043577 32667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:57:06.044878 32667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:57:06.045003 32667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:06.045136 32667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:57:06.045222 32667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:06.079425 32667 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:36647
I20260927 13:57:06.079481 325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:36647 every 8 connection(s)
I20260927 13:57:06.080772 32667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:57:06.088774 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:06.083353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:32913 (local address 127.22.26.254:42363)
0927 13:57:06.083526 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:06.083531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:06.084940 (+ 1409us) server_negotiation.cc:407] Connection header received
0927 13:57:06.086606 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:06.086610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:06.086679 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:06.086768 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:06.088099 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:06.088104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:06.088108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:06.088127 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:06.088164 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:06.088177 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:06.088320 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:06.088473 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 13:57:06.088528 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:57:06.089617 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:06.083868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42363 (local address 127.22.26.194:32913)
0927 13:57:06.084793 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:06.084837 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:06.086352 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:06.086959 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:06.086966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:06.087053 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:06.087787 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:57:06.087790 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:06.087827 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:57:06.087909 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:57:06.088289 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:57:06.088296 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:57:06.088301 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:57:06.088502 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 13:57:06.088703 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":269,"threads_started":1}
I20260927 13:57:06.089668 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 32667
I20260927 13:57:06.089861 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:57:06.090487 22635 external_mini_cluster.cc:1428] Setting key d91c439e9a38975159fc880c186ce66a
I20260927 13:57:06.091035 326 heartbeater.cc:347] Connected to a master server at 127.22.26.254:42363
I20260927 13:57:06.091424 326 heartbeater.cc:464] Registering TS with master...
I20260927 13:57:06.092069 326 heartbeater.cc:511] Master 127.22.26.254:42363 requested a full tablet report, sending...
I20260927 13:57:06.093537 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:42363
--builtin_ntp_servers=127.22.26.212:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 13:57:06.093804 32477 ts_manager.cc:194] Registered new tserver with Master: d8a89c53a82e441fad9b349bb853665b (127.22.26.194:36647)
W20260927 13:57:06.411949 329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:57:06.412213 329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:57:06.412305 329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:57:06.419282 329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:57:06.419370 329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:57:06.419472 329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:57:06.428275 329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:42363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:57:06.429450 329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:57:06.431167 329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:57:06.447574 334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:06.447548 335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:06.447788 337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:06.449260 329 server_base.cc:1034] running on GCE node
I20260927 13:57:06.449926 329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:57:06.450829 329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:57:06.452082 329 hybrid_clock.cc:648] HybridClock initialized: now 1790517426452018 us; error 74 us; skew 500 ppm
I20260927 13:57:06.452435 329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:57:06.454823 329 webserver.cc:460] Webserver started at http://127.22.26.195:44489/ using document root <none> and password file <none>
I20260927 13:57:06.455600 329 fs_manager.cc:359] Metadata directory not provided
I20260927 13:57:06.455724 329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:57:06.456043 329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:57:06.458870 329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "7d64835ed8524c1e8064aa199a6439f2"
format_stamp: "Formatted at 2026-09-27 13:57:06 on dist-test-slave-gkks"
server_key: "1dd45fe39a42ff9f977f58814e8173fc"
server_key_iv: "6f2e1957458d1ee9d001bd5db50d08cb"
server_key_version: "encryptionkey@0"
I20260927 13:57:06.459748 329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "7d64835ed8524c1e8064aa199a6439f2"
format_stamp: "Formatted at 2026-09-27 13:57:06 on dist-test-slave-gkks"
server_key: "1dd45fe39a42ff9f977f58814e8173fc"
server_key_iv: "6f2e1957458d1ee9d001bd5db50d08cb"
server_key_version: "encryptionkey@0"
I20260927 13:57:06.465122 329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:57:06.468809 343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:06.470199 329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:57:06.470381 329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "7d64835ed8524c1e8064aa199a6439f2"
format_stamp: "Formatted at 2026-09-27 13:57:06 on dist-test-slave-gkks"
server_key: "1dd45fe39a42ff9f977f58814e8173fc"
server_key_iv: "6f2e1957458d1ee9d001bd5db50d08cb"
server_key_version: "encryptionkey@0"
I20260927 13:57:06.470616 329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:57:06.476310 329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:57:06.477123 329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:57:06.477397 329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:57:06.478296 329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:57:06.479730 329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:57:06.479851 329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:06.479991 329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:57:06.480086 329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:06.512382 329 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:44755
I20260927 13:57:06.512429 455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:44755 every 8 connection(s)
I20260927 13:57:06.513700 329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:57:06.514827 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 329
I20260927 13:57:06.514990 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:57:06.515563 22635 external_mini_cluster.cc:1428] Setting key 37fe75c9b068d5b5bd5572ab64ab59d6
I20260927 13:57:06.522826 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:06.516463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:48685 (local address 127.22.26.254:42363)
0927 13:57:06.516668 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:06.516675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:06.518348 (+ 1673us) server_negotiation.cc:407] Connection header received
0927 13:57:06.520242 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:06.520246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:06.520314 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:06.520463 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:06.522096 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:06.522101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:06.522106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:06.522131 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:06.522162 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:06.522173 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:06.522295 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:06.522492 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 13:57:06.522561 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:57:06.523782 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:06.517121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42363 (local address 127.22.26.195:48685)
0927 13:57:06.518208 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:06.518256 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:06.519934 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:06.520712 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:06.520720 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:06.520810 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:06.521732 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:57:06.521735 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:06.521807 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:57:06.521937 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:57:06.522268 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:57:06.522279 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:57:06.522286 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 13:57:06.522579 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 13:57:06.522820 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":338,"threads_started":1}
I20260927 13:57:06.524955 456 heartbeater.cc:347] Connected to a master server at 127.22.26.254:42363
I20260927 13:57:06.525251 456 heartbeater.cc:464] Registering TS with master...
I20260927 13:57:06.525811 456 heartbeater.cc:511] Master 127.22.26.254:42363 requested a full tablet report, sending...
I20260927 13:57:06.527175 32477 ts_manager.cc:194] Registered new tserver with Master: 7d64835ed8524c1e8064aa199a6439f2 (127.22.26.195:44755)
I20260927 13:57:06.530570 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 13:57:06.536602 32665 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:57:06.535142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41862 (local address 127.22.26.254:42363)
0927 13:57:06.535286 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:06.535290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:06.535802 (+ 512us) server_negotiation.cc:407] Connection header received
0927 13:57:06.535904 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:06.535908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:06.535960 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:06.536037 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:06.536351 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41862: BlockingRecv error: recv got EOF from 127.0.0.1:41862 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 13:57:06.537032 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 32536
I20260927 13:57:06.540519 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:06.538015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.22.26.193:45959)
0927 13:57:06.538528 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:06.538534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:06.538560 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:57:06.538638 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:06.538644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:06.538832 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:06.538947 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:06.539902 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:06.539909 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:06.539919 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:06.539937 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:06.539979 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:06.539993 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:06.540098 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:06.540188 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 13:57:06.540300 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
I20260927 13:57:06.541735 32658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:57:06.626178 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 32536
I20260927 13:57:06.642081 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 32667
I20260927 13:57:06.645506 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:06.643116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.22.26.194:36647)
0927 13:57:06.643615 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:06.643622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:06.643660 (+ 38us) server_negotiation.cc:407] Connection header received
0927 13:57:06.643723 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:06.643729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:06.643941 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:06.644065 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:06.644936 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:06.644942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:06.644955 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:06.644967 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:06.645012 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:06.645030 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:06.645133 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:06.645210 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 13:57:06.645290 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":223,"threads_started":1}
I20260927 13:57:06.646704 321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:57:06.725982 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 32667
I20260927 13:57:06.741446 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 329
I20260927 13:57:06.745239 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:06.742466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.22.26.195:44755)
0927 13:57:06.743129 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:06.743145 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:06.743177 (+ 32us) server_negotiation.cc:407] Connection header received
0927 13:57:06.743241 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:06.743247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:06.743470 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:06.743594 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:06.744569 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:06.744575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:06.744587 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:57:06.744602 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:06.744657 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:57:06.744672 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:06.744801 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:06.744894 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 13:57:06.744973 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":315,"threads_started":1}
I20260927 13:57:06.746544 451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:57:06.827389 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 329
I20260927 13:57:06.842911 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 32446
I20260927 13:57:06.844166 32507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:57:06.945856 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 32446
2026-09-27T13:57:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2299 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:57:07 dist-test-slave-gkks krb5kdc[474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:57:07 dist-test-slave-gkks krb5kdc[474](info): set up 2 sockets
Sep 27 13:57:07 dist-test-slave-gkks krb5kdc[474](info): commencing operation
krb5kdc: starting...
W20260927 13:57:08.040145 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:57:08 dist-test-slave-gkks krb5kdc[474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:57:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:57:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:57:08.252556 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:37089
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:37089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:57:08.569970 487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:57:08.570247 487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:57:08.570423 487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:57:08.577493 487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:57:08.577592 487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:57:08.577710 487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:57:08.577809 487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:57:08.577893 487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:57:08.587417 487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:37089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:37089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:57:08.588652 487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:57:08.589989 487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:57:08.601488 492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:08.601574 495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:08.601517 493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:08.602468 487 server_base.cc:1034] running on GCE node
I20260927 13:57:08.603317 487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:57:08.604763 487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:57:08.605993 487 hybrid_clock.cc:648] HybridClock initialized: now 1790517428605958 us; error 46 us; skew 500 ppm
I20260927 13:57:08.606376 487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:57:08 dist-test-slave-gkks krb5kdc[474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517428, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:57:08.609375 487 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:57:08.610755 487 webserver.cc:460] Webserver started at http://127.22.26.254:39629/ using document root <none> and password file <none>
I20260927 13:57:08.611474 487 fs_manager.cc:359] Metadata directory not provided
I20260927 13:57:08.611618 487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:57:08.611909 487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:57:08.614714 487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "3bb54e82950248219c338d717e77073a"
format_stamp: "Formatted at 2026-09-27 13:57:08 on dist-test-slave-gkks"
server_key: "6d689a5e885e336a4cb5cc7fa53ca1d8"
server_key_iv: "b9bb32b1811d4be86c338d99c786a45f"
server_key_version: "encryptionkey@0"
I20260927 13:57:08.615481 487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "3bb54e82950248219c338d717e77073a"
format_stamp: "Formatted at 2026-09-27 13:57:08 on dist-test-slave-gkks"
server_key: "6d689a5e885e336a4cb5cc7fa53ca1d8"
server_key_iv: "b9bb32b1811d4be86c338d99c786a45f"
server_key_version: "encryptionkey@0"
I20260927 13:57:08.620311 487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 13:57:08.623764 502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:08.625066 487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:57:08.625258 487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "3bb54e82950248219c338d717e77073a"
format_stamp: "Formatted at 2026-09-27 13:57:08 on dist-test-slave-gkks"
server_key: "6d689a5e885e336a4cb5cc7fa53ca1d8"
server_key_iv: "b9bb32b1811d4be86c338d99c786a45f"
server_key_version: "encryptionkey@0"
I20260927 13:57:08.625448 487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:57:08.638087 487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:57:08.641903 487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:57:08.642177 487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:57:08.659116 487 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:37089
I20260927 13:57:08.659122 553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:37089 every 8 connection(s)
I20260927 13:57:08.660482 487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:57:08.661104 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 487
I20260927 13:57:08.661320 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:57:08.661952 22635 external_mini_cluster.cc:1428] Setting key 4742b074a2741940669fe6558f168bf2
I20260927 13:57:08.664909 554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:57:08 dist-test-slave-gkks krb5kdc[474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:57:08.677963 554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a: Bootstrap starting.
I20260927 13:57:08.681948 554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a: Neither blocks nor log segments found. Creating new log.
I20260927 13:57:08.683287 554 log.cc:824] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a: Log is configured to *not* fsync() on all Append() calls
I20260927 13:57:08.685710 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:08.663927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.22.26.254:37089)
0927 13:57:08.664681 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:08.664691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:08.664729 (+ 38us) server_negotiation.cc:407] Connection header received
0927 13:57:08.666336 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:08.666356 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:08.666699 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:08.666996 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:08.668817 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:08.670124 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:08.671973 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:08.672608 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:08.672733 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:08.677271 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:08.677283 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:08.677302 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:57:08.677347 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:08.681206 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:08.681972 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:08.681981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:08.681988 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:08.682193 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:08.682554 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:08.682558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:08.682561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:08.683281 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:08.683589 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:08.684034 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 13:57:08.684482 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":353,"threads_started":1}
I20260927 13:57:08.686699 554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a: No bootstrap required, opened a new log
I20260927 13:57:08.694314 554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb54e82950248219c338d717e77073a" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37089 } }
I20260927 13:57:08.694634 554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:57:08.694705 554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb54e82950248219c338d717e77073a, State: Initialized, Role: FOLLOWER
I20260927 13:57:08.695219 554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb54e82950248219c338d717e77073a" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37089 } }
I20260927 13:57:08.695416 554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:57:08.695508 554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:57:08.695660 554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:57:08.697942 554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb54e82950248219c338d717e77073a" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37089 } }
I20260927 13:57:08.698365 554 leader_election.cc:302] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bb54e82950248219c338d717e77073a; no voters:
I20260927 13:57:08.698887 554 leader_election.cc:288] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:57:08.699039 559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:57:08.699816 559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [term 1 LEADER]: Becoming Leader. State: Replica: 3bb54e82950248219c338d717e77073a, State: Running, Role: LEADER
I20260927 13:57:08.700359 559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb54e82950248219c338d717e77073a" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37089 } }
I20260927 13:57:08.700990 554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:57:08.702980 561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bb54e82950248219c338d717e77073a. Latest consensus state: current_term: 1 leader_uuid: "3bb54e82950248219c338d717e77073a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb54e82950248219c338d717e77073a" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37089 } } }
I20260927 13:57:08.703400 561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [sys.catalog]: This master's current role is: LEADER
I20260927 13:57:08.703013 560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bb54e82950248219c338d717e77073a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb54e82950248219c338d717e77073a" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37089 } } }
I20260927 13:57:08.703723 560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a [sys.catalog]: This master's current role is: LEADER
I20260927 13:57:08.705982 564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:57:08.712620 564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:57:08.722411 564 catalog_manager.cc:1347] Generated new cluster ID: b8271e282aea4b8487b26c9b01a70256
I20260927 13:57:08.722513 564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:57:08.742462 564 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:57:08.743897 564 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:57:08.750361 564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a: Generated new TSK 0
I20260927 13:57:08.751313 564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:57:08.816915 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:37089
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:57:09.136062 580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:57:09.136365 580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:57:09.136534 580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:57:09.143536 580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:57:09.143678 580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:57:09.143841 580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:57:09.153194 580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:37089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:57:09.154393 580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:57:09.155658 580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:57:09.168434 585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:09.168403 588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:09.168370 586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:09.169575 580 server_base.cc:1034] running on GCE node
I20260927 13:57:09.170183 580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:57:09.171044 580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:57:09.172294 580 hybrid_clock.cc:648] HybridClock initialized: now 1790517429172261 us; error 32 us; skew 500 ppm
I20260927 13:57:09.172605 580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:57:09 dist-test-slave-gkks krb5kdc[474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517429, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:57:09.175601 580 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:57:09.177029 580 webserver.cc:460] Webserver started at http://127.22.26.193:45121/ using document root <none> and password file <none>
I20260927 13:57:09.177695 580 fs_manager.cc:359] Metadata directory not provided
I20260927 13:57:09.177840 580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:57:09.178138 580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:57:09.181033 580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "3a9f9388f5ee450da589d5230caabe05"
format_stamp: "Formatted at 2026-09-27 13:57:09 on dist-test-slave-gkks"
server_key: "a8159da9a06313d99cbd51ab0ace3087"
server_key_iv: "222e75c74281657c1ed12d0c526c5467"
server_key_version: "encryptionkey@0"
I20260927 13:57:09.181771 580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "3a9f9388f5ee450da589d5230caabe05"
format_stamp: "Formatted at 2026-09-27 13:57:09 on dist-test-slave-gkks"
server_key: "a8159da9a06313d99cbd51ab0ace3087"
server_key_iv: "222e75c74281657c1ed12d0c526c5467"
server_key_version: "encryptionkey@0"
I20260927 13:57:09.186756 580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:57:09.190021 595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:09.191337 580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:57:09.191499 580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "3a9f9388f5ee450da589d5230caabe05"
format_stamp: "Formatted at 2026-09-27 13:57:09 on dist-test-slave-gkks"
server_key: "a8159da9a06313d99cbd51ab0ace3087"
server_key_iv: "222e75c74281657c1ed12d0c526c5467"
server_key_version: "encryptionkey@0"
I20260927 13:57:09.191679 580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:57:09.211061 580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:57:09.215411 580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:57:09.215710 580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:57:09.216737 580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:57:09.218075 580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:57:09.218158 580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:09.218243 580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:57:09.218287 580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:09.251667 580 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:40657
I20260927 13:57:09.251710 707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:40657 every 8 connection(s)
I20260927 13:57:09.252939 580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:57:09.261425 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 580
I20260927 13:57:09.261656 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:57:09.262310 22635 external_mini_cluster.cc:1428] Setting key 823fb7838a4939f3b6977b8120e41aad
Sep 27 13:57:09 dist-test-slave-gkks krb5kdc[474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517429, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:57:09.274382 710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:09.255487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:38535 (local address 127.22.26.254:37089)
0927 13:57:09.255894 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:09.255899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:09.256917 (+ 1018us) server_negotiation.cc:407] Connection header received
0927 13:57:09.258736 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:09.258741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:09.258832 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:09.258975 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:09.260548 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:09.261584 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:09.263454 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:09.263874 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:09.264028 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:09.267550 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:09.267559 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:09.267567 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:57:09.267613 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:09.271212 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:09.271930 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:09.271938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:09.271943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:09.272097 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:09.272497 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:09.272504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:09.272509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:09.272896 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:09.273049 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:09.273894 (+ 845us) server_negotiation.cc:299] Negotiation successful
0927 13:57:09.274127 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":210,"threads_started":1}
I20260927 13:57:09.275857 711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:09.256059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:37089 (local address 127.22.26.193:38535)
0927 13:57:09.256759 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:09.256798 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:09.258452 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:09.259262 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:09.259271 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:09.259789 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:09.260339 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:09.260357 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:09.261789 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:09.261795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:09.263280 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:09.263293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:09.264169 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:09.264177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:09.264441 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:09.265141 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:57:09.265168 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:09.267326 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:57:09.271344 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:57:09.271352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:57:09.271369 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:57:09.271788 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:57:09.272206 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:57:09.272209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:57:09.272212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:57:09.272380 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:57:09.273049 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:57:09.273059 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:57:09.273505 (+ 446us) client_negotiation.cc:769] Sending connection context
0927 13:57:09.273798 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 13:57:09.274380 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":268,"threads_started":1}
I20260927 13:57:09.277400 708 heartbeater.cc:347] Connected to a master server at 127.22.26.254:37089
I20260927 13:57:09.277714 708 heartbeater.cc:464] Registering TS with master...
I20260927 13:57:09.278429 708 heartbeater.cc:511] Master 127.22.26.254:37089 requested a full tablet report, sending...
I20260927 13:57:09.280495 519 ts_manager.cc:194] Registered new tserver with Master: 3a9f9388f5ee450da589d5230caabe05 (127.22.26.193:40657)
I20260927 13:57:09.281941 519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:38535
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:57:09.315958 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:37089
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:57:09.634635 714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:57:09.634936 714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:57:09.635157 714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:57:09.642180 714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:57:09.642319 714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:57:09.642513 714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:57:09.652122 714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:37089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:57:09.653344 714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:57:09.654616 714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:57:09.667558 719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:09.667480 720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:09.668220 722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:09.669178 714 server_base.cc:1034] running on GCE node
I20260927 13:57:09.669817 714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:57:09.670908 714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:57:09.672137 714 hybrid_clock.cc:648] HybridClock initialized: now 1790517429672075 us; error 54 us; skew 500 ppm
I20260927 13:57:09.672490 714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:57:09 dist-test-slave-gkks krb5kdc[474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517429, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:57:09.676301 714 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:57:09.677870 714 webserver.cc:460] Webserver started at http://127.22.26.194:38353/ using document root <none> and password file <none>
I20260927 13:57:09.678678 714 fs_manager.cc:359] Metadata directory not provided
I20260927 13:57:09.678815 714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:57:09.679215 714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:57:09.682008 714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "c163bc0a32584bcf9aa801b9b95ab384"
format_stamp: "Formatted at 2026-09-27 13:57:09 on dist-test-slave-gkks"
server_key: "20a3fa176584d186bb28864d5e71e488"
server_key_iv: "d74f831cc4edafbb9d6a1e05a5918843"
server_key_version: "encryptionkey@0"
I20260927 13:57:09.682751 714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "c163bc0a32584bcf9aa801b9b95ab384"
format_stamp: "Formatted at 2026-09-27 13:57:09 on dist-test-slave-gkks"
server_key: "20a3fa176584d186bb28864d5e71e488"
server_key_iv: "d74f831cc4edafbb9d6a1e05a5918843"
server_key_version: "encryptionkey@0"
I20260927 13:57:09.687805 714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 13:57:09.691273 729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:09.692539 714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:57:09.692761 714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "c163bc0a32584bcf9aa801b9b95ab384"
format_stamp: "Formatted at 2026-09-27 13:57:09 on dist-test-slave-gkks"
server_key: "20a3fa176584d186bb28864d5e71e488"
server_key_iv: "d74f831cc4edafbb9d6a1e05a5918843"
server_key_version: "encryptionkey@0"
I20260927 13:57:09.692950 714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:57:09.721086 714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:57:09.725064 714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:57:09.725363 714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:57:09.726305 714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:57:09.727694 714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:57:09.727797 714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:09.727941 714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:57:09.728015 714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:09.761579 714 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:33051
I20260927 13:57:09.761669 841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:33051 every 8 connection(s)
I20260927 13:57:09.762866 714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:57:09.767472 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 714
I20260927 13:57:09.767661 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:57:09.768097 22635 external_mini_cluster.cc:1428] Setting key 0a89d03d4faefbac9102ac67745bcea2
Sep 27 13:57:09 dist-test-slave-gkks krb5kdc[474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517429, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:57:09.784861 710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:09.765616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:53275 (local address 127.22.26.254:37089)
0927 13:57:09.765794 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:09.765799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:09.766920 (+ 1121us) server_negotiation.cc:407] Connection header received
0927 13:57:09.769008 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:09.769012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:09.769108 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:09.769230 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:09.770732 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:09.771940 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:09.773408 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:09.774137 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:09.774286 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:09.777997 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:09.778005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:09.778011 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:57:09.778065 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:09.781565 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:09.782234 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:09.782242 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:09.782249 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:09.782413 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:09.782968 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:09.782976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:09.782982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:09.783386 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:09.783529 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:09.784168 (+ 639us) server_negotiation.cc:299] Negotiation successful
0927 13:57:09.784413 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:57:09.785466 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:09.766062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:37089 (local address 127.22.26.194:53275)
0927 13:57:09.766757 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:09.766803 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:09.768691 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:09.769418 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:09.769426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:09.770011 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:09.770557 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:09.770576 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:09.772074 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:09.772077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:09.773260 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:09.773276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:09.774333 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:09.774337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:09.774526 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:09.775238 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:57:09.775259 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:09.777777 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:57:09.781697 (+ 3920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:57:09.781703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:57:09.781714 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:57:09.782100 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:57:09.782541 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:57:09.782548 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:57:09.782553 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:57:09.782837 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:57:09.783518 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:57:09.783524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:57:09.783805 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 13:57:09.784022 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 13:57:09.784380 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":280,"threads_started":1}
I20260927 13:57:09.787009 842 heartbeater.cc:347] Connected to a master server at 127.22.26.254:37089
I20260927 13:57:09.787360 842 heartbeater.cc:464] Registering TS with master...
I20260927 13:57:09.788244 842 heartbeater.cc:511] Master 127.22.26.254:37089 requested a full tablet report, sending...
I20260927 13:57:09.789928 519 ts_manager.cc:194] Registered new tserver with Master: c163bc0a32584bcf9aa801b9b95ab384 (127.22.26.194:33051)
I20260927 13:57:09.790724 519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:53275
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:57:09.822290 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:37089
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:57:10.137837 847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:57:10.138094 847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:57:10.138249 847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:57:10.145244 847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:57:10.145337 847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:57:10.145534 847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:57:10.154660 847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:37089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:57:10.155850 847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:57:10.157085 847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:57:10.169895 853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:10.169942 855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:10.169823 852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:10.170748 847 server_base.cc:1034] running on GCE node
I20260927 13:57:10.171358 847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:57:10.172194 847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:57:10.173383 847 hybrid_clock.cc:648] HybridClock initialized: now 1790517430173325 us; error 50 us; skew 500 ppm
I20260927 13:57:10.173688 847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:57:10 dist-test-slave-gkks krb5kdc[474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517430, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:57:10.176762 847 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:57:10.178205 847 webserver.cc:460] Webserver started at http://127.22.26.195:41071/ using document root <none> and password file <none>
I20260927 13:57:10.178859 847 fs_manager.cc:359] Metadata directory not provided
I20260927 13:57:10.178987 847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:57:10.179318 847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:57:10.182111 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "5034978d78504fd888b784c22e9f837e"
format_stamp: "Formatted at 2026-09-27 13:57:10 on dist-test-slave-gkks"
server_key: "91e145f8ddfebc2a7b1aba7c2b70429a"
server_key_iv: "0ea16b1d6077facbc96fcd8e100425a6"
server_key_version: "encryptionkey@0"
I20260927 13:57:10.182847 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "5034978d78504fd888b784c22e9f837e"
format_stamp: "Formatted at 2026-09-27 13:57:10 on dist-test-slave-gkks"
server_key: "91e145f8ddfebc2a7b1aba7c2b70429a"
server_key_iv: "0ea16b1d6077facbc96fcd8e100425a6"
server_key_version: "encryptionkey@0"
I20260927 13:57:10.187698 847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:57:10.191007 862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:10.192397 847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:57:10.192521 847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "5034978d78504fd888b784c22e9f837e"
format_stamp: "Formatted at 2026-09-27 13:57:10 on dist-test-slave-gkks"
server_key: "91e145f8ddfebc2a7b1aba7c2b70429a"
server_key_iv: "0ea16b1d6077facbc96fcd8e100425a6"
server_key_version: "encryptionkey@0"
I20260927 13:57:10.192682 847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:57:10.217473 847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:57:10.221283 847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:57:10.221517 847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:57:10.222373 847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:57:10.223610 847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:57:10.223681 847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:10.223760 847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:57:10.223804 847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:10.257735 847 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:39155
I20260927 13:57:10.257781 974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:39155 every 8 connection(s)
I20260927 13:57:10.258893 847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:57:10.261989 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 847
I20260927 13:57:10.262235 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:57:10.262759 22635 external_mini_cluster.cc:1428] Setting key bbcb6fd2f7d4960051309056015a68b0
Sep 27 13:57:10 dist-test-slave-gkks krb5kdc[474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517430, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:57:10.279026 710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.261302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:43437 (local address 127.22.26.254:37089)
0927 13:57:10.261482 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:10.261487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:10.262878 (+ 1391us) server_negotiation.cc:407] Connection header received
0927 13:57:10.264788 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:10.264793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:10.264909 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:10.264991 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:10.266603 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.267469 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.268954 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.269350 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.269502 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.272863 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:10.272868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:10.272871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:57:10.272903 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:10.275616 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:10.276435 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:10.276439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:10.276441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:10.276575 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:10.277082 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:10.277088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:10.277091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:10.277480 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:10.277653 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:10.278594 (+ 941us) server_negotiation.cc:299] Negotiation successful
0927 13:57:10.278754 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:57:10.279955 977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.261751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:37089 (local address 127.22.26.195:43437)
0927 13:57:10.262673 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:10.262732 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:10.264483 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:10.265262 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:10.265275 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:10.265879 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:57:10.266412 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:10.266429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.267613 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.267617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:10.268798 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:10.268810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.269545 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.269550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:10.269739 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.270365 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:57:10.270387 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:57:10.272650 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:57:10.275761 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:57:10.275768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:57:10.275779 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:57:10.276310 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:57:10.276696 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:57:10.276702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:57:10.276706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:57:10.276945 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:57:10.277669 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:57:10.277679 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:57:10.278077 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 13:57:10.278420 (+ 343us) client_negotiation.cc:240] Negotiation successful
0927 13:57:10.278853 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":359,"threads_started":1}
I20260927 13:57:10.281283 975 heartbeater.cc:347] Connected to a master server at 127.22.26.254:37089
I20260927 13:57:10.281574 975 heartbeater.cc:464] Registering TS with master...
I20260927 13:57:10.282269 975 heartbeater.cc:511] Master 127.22.26.254:37089 requested a full tablet report, sending...
I20260927 13:57:10.283663 519 ts_manager.cc:194] Registered new tserver with Master: 5034978d78504fd888b784c22e9f837e (127.22.26.195:39155)
I20260927 13:57:10.284418 519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:43437
I20260927 13:57:10.284865 708 heartbeater.cc:503] Master 127.22.26.254:37089 was elected leader, sending a full tablet report...
I20260927 13:57:10.290632 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:57:10.307976 710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.295048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.22.26.254:37089)
0927 13:57:10.295283 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:10.295293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:10.295825 (+ 532us) server_negotiation.cc:407] Connection header received
0927 13:57:10.296029 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:10.296034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:10.296118 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:10.296217 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:10.297749 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.298468 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.299757 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.300174 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.300310 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.302746 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:10.302750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:10.302753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:57:10.302783 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:10.305384 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:10.306007 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:10.306015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:10.306021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:10.306148 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:10.306526 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:10.306532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:10.306536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:10.306854 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:10.307156 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:10.307472 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 13:57:10.307687 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 13:57:10.311877 519 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:57:10.318624 519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:57:10.344259 987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.331136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:39155 (local address 127.0.0.1:34380)
0927 13:57:10.333193 (+ 2057us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:10.333212 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:10.333330 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:10.334782 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:10.334789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:10.334826 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:57:10.335444 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:10.335461 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.338304 (+ 2843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.338313 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:10.341356 (+ 3043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:10.341372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.343215 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.343222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:10.343424 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.343480 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 13:57:10.343620 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 13:57:10.343811 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1948,"mutex_wait_us":1,"thread_start_us":2292,"threads_started":1}
I20260927 13:57:10.344846 989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.331184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.22.26.194:33051)
0927 13:57:10.331985 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:10.331995 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:10.333176 (+ 1181us) server_negotiation.cc:407] Connection header received
0927 13:57:10.333298 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:10.333309 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:10.333570 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:10.334101 (+ 531us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:57:10.335416 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.337558 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.341702 (+ 4144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.343670 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.343806 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.344004 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:10.344358 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 13:57:10.344510 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":848,"threads_started":1}
I20260927 13:57:10.344841 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.331756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.22.26.195:39155)
0927 13:57:10.334091 (+ 2335us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:10.334102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:10.334142 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:57:10.334255 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:10.334265 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:10.334529 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:10.335691 (+ 1162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:57:10.336044 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.338107 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.341995 (+ 3888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.342960 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.343139 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.343374 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:10.344139 (+ 765us) server_negotiation.cc:299] Negotiation successful
0927 13:57:10.344328 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2107,"thread_start_us":2184,"threads_started":1}
I20260927 13:57:10.345867 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.331116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:33051 (local address 127.0.0.1:41424)
0927 13:57:10.332621 (+ 1505us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:10.332674 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:10.332910 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:10.334494 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:10.334506 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:10.334556 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:57:10.335142 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:10.335161 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.337774 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.337783 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:10.341462 (+ 3679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:10.341481 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.343966 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.343973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:10.344158 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.344188 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:57:10.345358 (+ 1170us) client_negotiation.cc:240] Negotiation successful
0927 13:57:10.345481 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"mutex_wait_us":366,"thread_start_us":1034,"threads_started":1}
I20260927 13:57:10.348412 910 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee326c206d6494da155b7d198122d1e (DEFAULT_TABLE table=test-table [id=7b5c2476b489462fbe88bca9fe16af91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:57:10.349212 777 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee326c206d6494da155b7d198122d1e (DEFAULT_TABLE table=test-table [id=7b5c2476b489462fbe88bca9fe16af91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:57:10.349936 910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee326c206d6494da155b7d198122d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:57:10.350549 777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee326c206d6494da155b7d198122d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:57:10.351945 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.333405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40657 (local address 127.0.0.1:55692)
0927 13:57:10.341732 (+ 8327us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:10.341750 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:10.341850 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:10.344764 (+ 2914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:10.344768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:10.344794 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:57:10.345193 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:10.345205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.347367 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.347373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:10.349361 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:10.349376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.351212 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.351219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:10.351383 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.351414 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 13:57:10.351511 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 13:57:10.351660 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8189,"thread_start_us":7757,"threads_started":1}
I20260927 13:57:10.352043 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.333730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.22.26.193:40657)
0927 13:57:10.343905 (+ 10175us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:10.343915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:10.343945 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:57:10.344045 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:10.344052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:10.344280 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:10.345373 (+ 1093us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:57:10.345642 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.347221 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.349546 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.350973 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.351142 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.351345 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:10.351568 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 13:57:10.351737 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10025,"thread_start_us":9871,"threads_started":1}
I20260927 13:57:10.355742 643 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee326c206d6494da155b7d198122d1e (DEFAULT_TABLE table=test-table [id=7b5c2476b489462fbe88bca9fe16af91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:57:10.357134 643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee326c206d6494da155b7d198122d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:57:10.362282 993 tablet_bootstrap.cc:492] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384: Bootstrap starting.
I20260927 13:57:10.365456 994 tablet_bootstrap.cc:492] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e: Bootstrap starting.
I20260927 13:57:10.365793 993 tablet_bootstrap.cc:654] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384: Neither blocks nor log segments found. Creating new log.
I20260927 13:57:10.367122 993 log.cc:824] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384: Log is configured to *not* fsync() on all Append() calls
I20260927 13:57:10.369874 994 tablet_bootstrap.cc:654] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e: Neither blocks nor log segments found. Creating new log.
I20260927 13:57:10.371450 994 log.cc:824] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e: Log is configured to *not* fsync() on all Append() calls
I20260927 13:57:10.371533 996 tablet_bootstrap.cc:492] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05: Bootstrap starting.
I20260927 13:57:10.374804 993 tablet_bootstrap.cc:492] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384: No bootstrap required, opened a new log
I20260927 13:57:10.375176 993 ts_tablet_manager.cc:1397] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260927 13:57:10.375375 996 tablet_bootstrap.cc:654] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05: Neither blocks nor log segments found. Creating new log.
I20260927 13:57:10.375535 994 tablet_bootstrap.cc:492] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e: No bootstrap required, opened a new log
I20260927 13:57:10.375808 994 ts_tablet_manager.cc:1397] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 13:57:10.376757 996 log.cc:824] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05: Log is configured to *not* fsync() on all Append() calls
I20260927 13:57:10.380491 996 tablet_bootstrap.cc:492] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05: No bootstrap required, opened a new log
I20260927 13:57:10.380752 996 ts_tablet_manager.cc:1397] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 13:57:10.383391 993 raft_consensus.cc:348] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c163bc0a32584bcf9aa801b9b95ab384" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33051 } } peers { permanent_uuid: "5034978d78504fd888b784c22e9f837e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39155 } } peers { permanent_uuid: "3a9f9388f5ee450da589d5230caabe05" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40657 } }
I20260927 13:57:10.383783 993 raft_consensus.cc:374] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:57:10.383888 993 raft_consensus.cc:729] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c163bc0a32584bcf9aa801b9b95ab384, State: Initialized, Role: FOLLOWER
I20260927 13:57:10.384680 993 consensus_queue.cc:260] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c163bc0a32584bcf9aa801b9b95ab384" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33051 } } peers { permanent_uuid: "5034978d78504fd888b784c22e9f837e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39155 } } peers { permanent_uuid: "3a9f9388f5ee450da589d5230caabe05" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40657 } }
I20260927 13:57:10.385815 842 heartbeater.cc:503] Master 127.22.26.254:37089 was elected leader, sending a full tablet report...
I20260927 13:57:10.386523 993 ts_tablet_manager.cc:1428] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260927 13:57:10.386428 994 raft_consensus.cc:348] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c163bc0a32584bcf9aa801b9b95ab384" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33051 } } peers { permanent_uuid: "5034978d78504fd888b784c22e9f837e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39155 } } peers { permanent_uuid: "3a9f9388f5ee450da589d5230caabe05" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40657 } }
I20260927 13:57:10.386829 994 raft_consensus.cc:374] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:57:10.386948 994 raft_consensus.cc:729] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5034978d78504fd888b784c22e9f837e, State: Initialized, Role: FOLLOWER
I20260927 13:57:10.387565 994 consensus_queue.cc:260] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c163bc0a32584bcf9aa801b9b95ab384" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33051 } } peers { permanent_uuid: "5034978d78504fd888b784c22e9f837e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39155 } } peers { permanent_uuid: "3a9f9388f5ee450da589d5230caabe05" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40657 } }
I20260927 13:57:10.392145 994 ts_tablet_manager.cc:1428] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260927 13:57:10.392243 975 heartbeater.cc:503] Master 127.22.26.254:37089 was elected leader, sending a full tablet report...
I20260927 13:57:10.393337 996 raft_consensus.cc:348] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c163bc0a32584bcf9aa801b9b95ab384" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33051 } } peers { permanent_uuid: "5034978d78504fd888b784c22e9f837e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39155 } } peers { permanent_uuid: "3a9f9388f5ee450da589d5230caabe05" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40657 } }
I20260927 13:57:10.393666 996 raft_consensus.cc:374] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:57:10.393759 996 raft_consensus.cc:729] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9f9388f5ee450da589d5230caabe05, State: Initialized, Role: FOLLOWER
I20260927 13:57:10.394376 996 consensus_queue.cc:260] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c163bc0a32584bcf9aa801b9b95ab384" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33051 } } peers { permanent_uuid: "5034978d78504fd888b784c22e9f837e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39155 } } peers { permanent_uuid: "3a9f9388f5ee450da589d5230caabe05" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40657 } }
I20260927 13:57:10.395937 996 ts_tablet_manager.cc:1428] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 13:57:10.425041 999 raft_consensus.cc:482] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:57:10.425345 999 raft_consensus.cc:504] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c163bc0a32584bcf9aa801b9b95ab384" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33051 } } peers { permanent_uuid: "5034978d78504fd888b784c22e9f837e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39155 } } peers { permanent_uuid: "3a9f9388f5ee450da589d5230caabe05" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40657 } }
I20260927 13:57:10.427043 999 leader_election.cc:288] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5034978d78504fd888b784c22e9f837e (127.22.26.195:39155), 3a9f9388f5ee450da589d5230caabe05 (127.22.26.193:40657)
I20260927 13:57:10.433772 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.427498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:40951 (local address 127.22.26.195:39155)
0927 13:57:10.427695 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:10.427702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:10.428309 (+ 607us) server_negotiation.cc:407] Connection header received
0927 13:57:10.428583 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:10.428587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:10.428678 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:10.428777 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:57:10.429335 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.430199 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.432048 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.432795 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.432966 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.433049 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:10.433393 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 13:57:10.433520 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 13:57:10.433859 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.427617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:60485 (local address 127.22.26.193:40657)
0927 13:57:10.427818 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:10.427824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:10.428497 (+ 673us) server_negotiation.cc:407] Connection header received
0927 13:57:10.428750 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:10.428762 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:10.428866 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:10.428969 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:57:10.429560 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.430382 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.432129 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.432902 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.433077 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.433154 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:10.433485 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 13:57:10.433608 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:57:10.433933 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.427338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:39155 (local address 127.22.26.194:40951)
0927 13:57:10.428169 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:10.428204 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:10.428411 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:10.428816 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:10.428820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:10.428846 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:57:10.429190 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:10.429200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.430331 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.430338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:10.431891 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:10.431902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.433027 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.433033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:10.433234 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.433263 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 13:57:10.433369 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 13:57:10.433492 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":47,"thread_start_us":403,"threads_started":1}
I20260927 13:57:10.433990 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.427651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40657 (local address 127.22.26.194:60485)
0927 13:57:10.428396 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 13:57:10.428410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:57:10.428550 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:57:10.428988 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:57:10.428992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:57:10.429020 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:57:10.429419 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:10.429428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.430527 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.430532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:10.431977 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:57:10.431989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.433138 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.433143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:57:10.433327 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.433348 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 13:57:10.433447 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 13:57:10.433564 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":29,"thread_start_us":388,"threads_started":1}
I20260927 13:57:10.434906 930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee326c206d6494da155b7d198122d1e" candidate_uuid: "c163bc0a32584bcf9aa801b9b95ab384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5034978d78504fd888b784c22e9f837e" is_pre_election: true
I20260927 13:57:10.434953 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee326c206d6494da155b7d198122d1e" candidate_uuid: "c163bc0a32584bcf9aa801b9b95ab384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9f9388f5ee450da589d5230caabe05" is_pre_election: true
I20260927 13:57:10.435475 930 raft_consensus.cc:2454] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c163bc0a32584bcf9aa801b9b95ab384 in term 0.
I20260927 13:57:10.435473 663 raft_consensus.cc:2454] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c163bc0a32584bcf9aa801b9b95ab384 in term 0.
I20260927 13:57:10.436138 733 leader_election.cc:302] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a9f9388f5ee450da589d5230caabe05, 5034978d78504fd888b784c22e9f837e, c163bc0a32584bcf9aa801b9b95ab384; no voters:
I20260927 13:57:10.436491 999 raft_consensus.cc:2781] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:57:10.436656 999 raft_consensus.cc:482] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:57:10.436770 999 raft_consensus.cc:3037] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:57:10.439153 999 raft_consensus.cc:504] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c163bc0a32584bcf9aa801b9b95ab384" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33051 } } peers { permanent_uuid: "5034978d78504fd888b784c22e9f837e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39155 } } peers { permanent_uuid: "3a9f9388f5ee450da589d5230caabe05" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40657 } }
I20260927 13:57:10.439725 999 leader_election.cc:288] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [CANDIDATE]: Term 1 election: Requested vote from peers 5034978d78504fd888b784c22e9f837e (127.22.26.195:39155), 3a9f9388f5ee450da589d5230caabe05 (127.22.26.193:40657)
I20260927 13:57:10.440145 930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee326c206d6494da155b7d198122d1e" candidate_uuid: "c163bc0a32584bcf9aa801b9b95ab384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5034978d78504fd888b784c22e9f837e"
I20260927 13:57:10.440150 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee326c206d6494da155b7d198122d1e" candidate_uuid: "c163bc0a32584bcf9aa801b9b95ab384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9f9388f5ee450da589d5230caabe05"
I20260927 13:57:10.440337 663 raft_consensus.cc:3037] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:57:10.440321 930 raft_consensus.cc:3037] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:57:10.442411 930 raft_consensus.cc:2454] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c163bc0a32584bcf9aa801b9b95ab384 in term 1.
I20260927 13:57:10.442417 663 raft_consensus.cc:2454] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c163bc0a32584bcf9aa801b9b95ab384 in term 1.
I20260927 13:57:10.442793 733 leader_election.cc:302] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5034978d78504fd888b784c22e9f837e, c163bc0a32584bcf9aa801b9b95ab384; no voters:
I20260927 13:57:10.443212 999 raft_consensus.cc:2781] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:57:10.443674 999 raft_consensus.cc:686] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [term 1 LEADER]: Becoming Leader. State: Replica: c163bc0a32584bcf9aa801b9b95ab384, State: Running, Role: LEADER
I20260927 13:57:10.444332 999 consensus_queue.cc:237] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c163bc0a32584bcf9aa801b9b95ab384" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33051 } } peers { permanent_uuid: "5034978d78504fd888b784c22e9f837e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39155 } } peers { permanent_uuid: "3a9f9388f5ee450da589d5230caabe05" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40657 } }
I20260927 13:57:10.448876 519 catalog_manager.cc:5495] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 reported cstate change: term changed from 0 to 1, leader changed from <none> to c163bc0a32584bcf9aa801b9b95ab384 (127.22.26.194). New cstate: current_term: 1 leader_uuid: "c163bc0a32584bcf9aa801b9b95ab384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c163bc0a32584bcf9aa801b9b95ab384" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5034978d78504fd888b784c22e9f837e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9f9388f5ee450da589d5230caabe05" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40657 } health_report { overall_health: UNKNOWN } } }
I20260927 13:57:10.473173 989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.468425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.22.26.194:33051)
0927 13:57:10.468659 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:10.468663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:10.468682 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:57:10.468874 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:10.468878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:10.468949 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:10.469064 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:57:10.469363 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.470033 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.471264 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.471653 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.471743 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.472159 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:57:10.472608 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:57:10.472711 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:10.472813 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 13:57:10.472871 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 13:57:10.481421 663 raft_consensus.cc:1261] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05 [term 1 FOLLOWER]: Refusing update from remote peer c163bc0a32584bcf9aa801b9b95ab384: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:57:10.481495 930 raft_consensus.cc:1261] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e [term 1 FOLLOWER]: Refusing update from remote peer c163bc0a32584bcf9aa801b9b95ab384: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:57:10.482403 1004 consensus_queue.cc:1035] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9f9388f5ee450da589d5230caabe05" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:57:10.482718 999 consensus_queue.cc:1035] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5034978d78504fd888b784c22e9f837e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:57:10.507742 519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:57:10.508031 519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:57:10.511708 519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bb54e82950248219c338d717e77073a: Sending DeleteTablet for 3 replicas of tablet 7ee326c206d6494da155b7d198122d1e
I20260927 13:57:10.512683 777 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee326c206d6494da155b7d198122d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:57:10 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:41424
I20260927 13:57:10.512795 910 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee326c206d6494da155b7d198122d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:57:10 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:34380
I20260927 13:57:10.513913 643 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee326c206d6494da155b7d198122d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:57:10 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:55692
I20260927 13:57:10.514783 1019 tablet_replica.cc:330] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05: stopping tablet replica
I20260927 13:57:10.514883 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 580
I20260927 13:57:10.515244 1019 raft_consensus.cc:2229] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:57:10.515750 1019 raft_consensus.cc:2258] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:57:10.518186 1019 ts_tablet_manager.cc:1905] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:57:10.527844 1019 ts_tablet_manager.cc:1918] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:57:10.528127 1019 log.cc:1196] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/7ee326c206d6494da155b7d198122d1e
I20260927 13:57:10.528906 1019 ts_tablet_manager.cc:1939] T 7ee326c206d6494da155b7d198122d1e P 3a9f9388f5ee450da589d5230caabe05: Deleting consensus metadata
I20260927 13:57:10.531179 504 catalog_manager.cc:4841] TS 3a9f9388f5ee450da589d5230caabe05 (127.22.26.193:40657): tablet 7ee326c206d6494da155b7d198122d1e (table test-table [id=7b5c2476b489462fbe88bca9fe16af91]) successfully deleted
I20260927 13:57:10.532490 1018 tablet_replica.cc:330] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e: stopping tablet replica
I20260927 13:57:10.533114 1018 raft_consensus.cc:2229] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:57:10.533782 1018 raft_consensus.cc:2258] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:57:10.537411 1018 ts_tablet_manager.cc:1905] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:57:10.541345 1017 tablet_replica.cc:330] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384: stopping tablet replica
I20260927 13:57:10.541793 1017 raft_consensus.cc:2229] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:57:10.542418 1017 raft_consensus.cc:2258] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:57:10.545830 1018 ts_tablet_manager.cc:1918] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:57:10.546023 1018 log.cc:1196] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/7ee326c206d6494da155b7d198122d1e
I20260927 13:57:10.546567 1018 ts_tablet_manager.cc:1939] T 7ee326c206d6494da155b7d198122d1e P 5034978d78504fd888b784c22e9f837e: Deleting consensus metadata
Sep 27 13:57:10 dist-test-slave-gkks krb5kdc[474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:57:10.548329 506 catalog_manager.cc:4841] TS 5034978d78504fd888b784c22e9f837e (127.22.26.195:39155): tablet 7ee326c206d6494da155b7d198122d1e (table test-table [id=7b5c2476b489462fbe88bca9fe16af91]) successfully deleted
I20260927 13:57:10.548591 1017 ts_tablet_manager.cc:1905] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:57:10.554896 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.517330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55694 (local address 127.22.26.193:40657)
0927 13:57:10.517499 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:10.517507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:10.531736 (+ 14229us) server_negotiation.cc:407] Connection header received
0927 13:57:10.531983 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:10.531989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:10.532071 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:10.532182 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:10.533979 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.537937 (+ 3958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.539687 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.540272 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.540359 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.549228 (+ 8869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:10.549243 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:10.549260 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:57:10.549304 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:10.552171 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:10.552850 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:10.552862 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:10.552865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:10.552983 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:10.553382 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:10.553386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:10.553389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:10.553721 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:10.554044 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:10.554300 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 13:57:10.554521 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:57:10.555217 1017 ts_tablet_manager.cc:1918] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:57:10.555392 1017 log.cc:1196] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/7ee326c206d6494da155b7d198122d1e
I20260927 13:57:10.555692 703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:57:10.556011 1017 ts_tablet_manager.cc:1939] T 7ee326c206d6494da155b7d198122d1e P c163bc0a32584bcf9aa801b9b95ab384: Deleting consensus metadata
I20260927 13:57:10.557859 505 catalog_manager.cc:4841] TS c163bc0a32584bcf9aa801b9b95ab384 (127.22.26.194:33051): tablet 7ee326c206d6494da155b7d198122d1e (table test-table [id=7b5c2476b489462fbe88bca9fe16af91]) successfully deleted
I20260927 13:57:10.644834 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 580
I20260927 13:57:10.670040 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 714
Sep 27 13:57:10 dist-test-slave-gkks krb5kdc[474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:57:10.687844 989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.671046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.22.26.194:33051)
0927 13:57:10.671317 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:10.671324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:10.671345 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:57:10.671542 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:10.671550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:10.671633 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:10.671717 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:10.673444 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.674453 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.676322 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.676759 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.676856 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.681879 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:10.681894 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:10.681908 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:57:10.681941 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:10.685094 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:10.685733 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:10.685740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:10.685745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:10.685883 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:10.686315 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:10.686320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:10.686330 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:10.686667 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:10.686996 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:10.687299 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 13:57:10.687517 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260927 13:57:10.688510 837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:57:10.803264 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 714
I20260927 13:57:10.822368 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 847
Sep 27 13:57:10 dist-test-slave-gkks krb5kdc[474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:57:10.837744 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:10.823159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34388 (local address 127.22.26.195:39155)
0927 13:57:10.823376 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:10.823380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:10.823398 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:57:10.823611 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:10.823616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:10.823688 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:10.823779 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:10.825047 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.825784 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.827061 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:10.827698 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:10.827803 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:10.832236 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:10.832250 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:10.832265 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:57:10.832303 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:10.834912 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:10.835697 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:10.835711 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:10.835713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:10.835850 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:10.836247 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:10.836251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:10.836253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:10.836590 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:10.836878 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:10.837221 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 13:57:10.837441 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 13:57:10.838375 970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:57:10.926617 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 847
I20260927 13:57:10.943601 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 487
I20260927 13:57:10.944753 549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:57:11.053519 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 487
2026-09-27T13:57:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4108 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:57:11 dist-test-slave-gkks krb5kdc[1027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:57:11 dist-test-slave-gkks krb5kdc[1027](info): set up 2 sockets
Sep 27 13:57:11 dist-test-slave-gkks krb5kdc[1027](info): commencing operation
krb5kdc: starting...
W20260927 13:57:12.168547 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:57:12 dist-test-slave-gkks krb5kdc[1027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:57:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:57:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/rangeradmin_127.22.26.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/rangeradmin_127.22.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/rangerlookup_127.22.26.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/rangerlookup_127.22.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/HTTP_127.22.26.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/HTTP_127.22.26.212@KRBTEST.COM.keytab.
I20260927 13:57:12.494319 22635 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUeHuEC/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/postgres -l logfile start
2026-09-27 13:57:15.110 UTC [1057] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 13:57:15.110 UTC [1057] LOG: listening on IPv4 address "127.22.26.212", port 40257
2026-09-27 13:57:15.113 UTC [1057] LOG: listening on Unix socket "/tmp/.s.PGSQL.40257"
2026-09-27 13:57:15.134 UTC [1060] LOG: database system was shut down at 2026-09-27 13:57:13 UTC
2026-09-27 13:57:15.138 UTC [1057] LOG: database system is ready to accept connections
I20260927 13:57:16.103919 22635 mini_postgres.cc:96] Postgres bound to 40257
2026-09-27 13:57:16.122 UTC [1068] FATAL: database "slave" does not exist
127.22.26.212:40257 - accepting connections
I20260927 13:57:16.123250 22635 mini_ranger.cc:161] Starting Ranger...
I20260927 13:57:16.152390 22635 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 13:57:16.613817 22635 mini_ranger.cc:87] Created ranger Postgres database
I20260927 13:57:16.613965 22635 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-admin
2026-09-27 13:57:17,296 [I] DB FLAVOR :POSTGRES
2026-09-27 13:57:17,297 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 13:57:17,297 [I] Checking connection..
2026-09-27 13:57:17,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 13:57:17,650 [I] Checking connection passed.
2026-09-27 13:57:17,650 [I] --------- Verifying version history table ---------
2026-09-27 13:57:17,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:57:17,878 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 13:57:17,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:57:18,118 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 13:57:18,119 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 13:57:18,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 13:57:18.343 UTC [1132] WARNING: there is no transaction in progress
2026-09-27 13:57:18,349 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 13:57:18,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:57:18,577 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 13:57:18,578 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 13:57:18,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 13:57:18,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 13:57:19,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gkks.c.gcp-upstream.internal','N') ;"
2026-09-27 13:57:19,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 13:57:19,543 [I] Table x_portal_user does not exist in database ranger
2026-09-27 13:57:19,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 13:57:19,775 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 13:57:19,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 13:57:20,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 13:57:20,226 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 13:57:20,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 13:57:21.158 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.173 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.189 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.203 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.217 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.235 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.286 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.293 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.307 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.317 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.327 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.341 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.350 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.357 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.363 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21.370 UTC [1240] WARNING: there is no transaction in progress
2026-09-27 13:57:21,983 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 13:57:21,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 13:57:22,231 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 13:57:22,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 13:57:22,488 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 13:57:22,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 13:57:22,747 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 13:57:22,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 13:57:23,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 13:57:23,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gkks.c.gcp-upstream.internal';"
2026-09-27 13:57:23,479 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 13:57:23,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 13:57:23,735 [I] DB_PATCHES have already been applied
I20260927 13:57:23.741459 22635 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUeHuEC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 13:57:23.741573 22635 mini_ranger.cc:193] Using host: 127.22.26.212
I20260927 13:57:23.756597 22635 mini_ranger.cc:227] Ranger admin URL: http://127.22.26.212:58719
Sep 27, 2026 1:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 1:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 1:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 1:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 1:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 1:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 1:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 13:57:26.037946 22635 mini_ranger.cc:160] Time spent starting Ranger: real 9.915s user 0.003s sys 0.042s
Sep 27, 2026 1:57:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 1:57:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 1:57:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 1:57:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 1:57:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 1:57:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 1:57:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 1:57:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 1:57:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 1:57:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 1:57:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 1:57:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 1:57:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 1:57:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 1:57:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 1:57:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 1:57:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 1:57:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 13:57:49.080214 22635 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/rangerkms_127.22.26.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/rangerkms_127.22.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.26.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.26.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.26.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/HTTP_127.22.26.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.26.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/HTTP_127.22.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 13:57:49 dist-test-slave-gkks krb5kdc[1027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517469, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 13:57:49.238901 22635 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 13:57:49.261687 22635 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 13:57:50.093387 22635 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 13:57:50.093521 22635 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-kms
I20260927 13:57:50.093672 22635 mini_ranger_kms.cc:227] Using postgres at 127.22.26.212:40257
2026-09-27 13:57:50,878 [I] DB FLAVOR :POSTGRES
2026-09-27 13:57:50,878 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 13:57:50,878 [I] Checking connection
2026-09-27 13:57:50,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 13:57:51,129 [I] connection success
2026-09-27 13:57:51,129 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 13:57:51,129 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 13:57:51,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 13:57:51,386 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 13:57:51,386 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 13:57:51,386 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 13:57:51,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:40257/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 13:57:51,662 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 13:57:52.144203 22635 mini_ranger_kms.cc:310] Created kms service
I20260927 13:57:52.351841 22635 mini_ranger_kms.cc:326] Created kudu user
I20260927 13:57:52.400596 22635 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 13:57:52 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517469, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:57:52.803632 22635 mini_ranger_kms.cc:384] Added ranger policy
I20260927 13:57:52.804009 22635 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 13:57:52.804080 22635 mini_ranger_kms.cc:240] Using host: 127.22.26.212
I20260927 13:57:52.826239 22635 mini_ranger_kms.cc:276] Ranger KMS PID: 1422
I20260927 13:57:52.826440 22635 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.26.212:54915
Sep 27, 2026 1:57:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 1:57:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 1:57:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 1:57:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 1:57:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 1:57:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 13:57:55.087545 22635 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.849s user 0.147s sys 0.372s
I20260927 13:57:55.087811 22635 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 13:57:55.088065 22635 mini_ranger_kms.cc:401] 127.22.26.212:54915/kms/v1/keys
Sep 27 13:57:55 dist-test-slave-gkks krb5kdc[1027](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790517475, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.26.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 13:57:55 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517475, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.26.212@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
Sep 27 13:57:56 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517475, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.26.212@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
Sep 27 13:57:58 dist-test-slave-gkks krb5kdc[1027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:57:58.311000 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:35121
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:35121
--ranger_config_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.26.212:54915/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:57:58.647383 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:57:58.647725 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:57:58.647883 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:57:58.655038 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:57:58.655231 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:57:58.655335 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:57:58.655441 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:57:58.655529 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:57:58.665644 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.26.212:54915/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:35121
--ranger_config_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:35121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:57:58.666925 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:57:58.668365 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:57:58.681264 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:58.681264 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:57:58.681680 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:57:58.683218 1472 server_base.cc:1034] running on GCE node
I20260927 13:57:58.684278 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:57:58.686172 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:57:58.687422 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790517478687367 us; error 39 us; skew 500 ppm
I20260927 13:57:58.687872 1472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:57:58 dist-test-slave-gkks krb5kdc[1027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517478, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:57:58.696318 1472 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:57:58.698233 1472 webserver.cc:460] Webserver started at http://127.22.26.254:40047/ using document root <none> and password file <none>
I20260927 13:57:58.699167 1472 fs_manager.cc:359] Metadata directory not provided
I20260927 13:57:58.699321 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:57:58.699695 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:57:58 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517478, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:57:58.752527 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "e605987c74f14d2ab879e9912335e26e"
format_stamp: "Formatted at 2026-09-27 13:57:58 on dist-test-slave-gkks"
server_key: "4ebd070443182b9373ca1e19045f88c6"
server_key_iv: "b500c6700660afca6d8dbe07aec48356"
server_key_version: "kuduclusterkey@0"
I20260927 13:57:58.753417 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "e605987c74f14d2ab879e9912335e26e"
format_stamp: "Formatted at 2026-09-27 13:57:58 on dist-test-slave-gkks"
server_key: "4ebd070443182b9373ca1e19045f88c6"
server_key_iv: "b500c6700660afca6d8dbe07aec48356"
server_key_version: "kuduclusterkey@0"
I20260927 13:57:58.758999 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:57:58.780418 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:57:58.782064 1472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 13:57:58.782258 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "e605987c74f14d2ab879e9912335e26e"
format_stamp: "Formatted at 2026-09-27 13:57:58 on dist-test-slave-gkks"
server_key: "4ebd070443182b9373ca1e19045f88c6"
server_key_iv: "b500c6700660afca6d8dbe07aec48356"
server_key_version: "kuduclusterkey@0"
I20260927 13:57:58.782505 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:57:58.799350 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:57:58.803632 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:57:58.803951 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:57:58.824451 1472 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:35121
I20260927 13:57:58.824550 1538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:35121 every 8 connection(s)
I20260927 13:57:58.825907 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:57:58.830688 1539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:57:58.830744 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 1472
I20260927 13:57:58.830936 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:57:58.831629 22635 external_mini_cluster.cc:1428] Setting key 64972d2e693201b959e034332e75a2ec
I20260927 13:57:58.846588 1539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e: Bootstrap starting.
Sep 27 13:57:58 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:57:58.850500 1539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e: Neither blocks nor log segments found. Creating new log.
I20260927 13:57:58.852003 1539 log.cc:824] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e: Log is configured to *not* fsync() on all Append() calls
I20260927 13:57:58.855445 1539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e: No bootstrap required, opened a new log
I20260927 13:57:58.858100 1542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:57:58.833679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.22.26.254:35121)
0927 13:57:58.834591 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0927 13:57:58.834607 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 13:57:58.834665 (+ 58us) server_negotiation.cc:407] Connection header received
0927 13:57:58.837819 (+ 3154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:57:58.837852 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:57:58.838408 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:57:58.839037 (+ 629us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:57:58.840156 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:58.842385 (+ 2229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:58.843789 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:57:58.844958 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:57:58.845151 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:57:58.849037 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:57:58.849049 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:57:58.849074 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:57:58.849121 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:57:58.853417 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:58.854126 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:58.854138 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:58.854149 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:58.854373 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:57:58.854754 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:57:58.854759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:57:58.854763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:57:58.855516 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:57:58.855823 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 13:57:58.856207 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 13:57:58.856630 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":374,"threads_started":1}
I20260927 13:57:58.865046 1539 raft_consensus.cc:348] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e605987c74f14d2ab879e9912335e26e" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35121 } }
I20260927 13:57:58.865409 1539 raft_consensus.cc:374] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:57:58.865480 1539 raft_consensus.cc:729] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e605987c74f14d2ab879e9912335e26e, State: Initialized, Role: FOLLOWER
I20260927 13:57:58.865978 1539 consensus_queue.cc:260] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e605987c74f14d2ab879e9912335e26e" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35121 } }
I20260927 13:57:58.866178 1539 raft_consensus.cc:388] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:57:58.866285 1539 raft_consensus.cc:482] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:57:58.866467 1539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:57:58.868793 1539 raft_consensus.cc:504] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e605987c74f14d2ab879e9912335e26e" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35121 } }
I20260927 13:57:58.869256 1539 leader_election.cc:302] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e605987c74f14d2ab879e9912335e26e; no voters:
I20260927 13:57:58.869918 1539 leader_election.cc:288] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:57:58.870066 1544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:57:58.870894 1544 raft_consensus.cc:686] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [term 1 LEADER]: Becoming Leader. State: Replica: e605987c74f14d2ab879e9912335e26e, State: Running, Role: LEADER
I20260927 13:57:58.871569 1544 consensus_queue.cc:237] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e605987c74f14d2ab879e9912335e26e" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35121 } }
I20260927 13:57:58.872568 1539 sys_catalog.cc:564] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:57:58.874917 1546 sys_catalog.cc:455] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e605987c74f14d2ab879e9912335e26e. Latest consensus state: current_term: 1 leader_uuid: "e605987c74f14d2ab879e9912335e26e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e605987c74f14d2ab879e9912335e26e" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35121 } } }
I20260927 13:57:58.874850 1545 sys_catalog.cc:455] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e605987c74f14d2ab879e9912335e26e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e605987c74f14d2ab879e9912335e26e" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35121 } } }
I20260927 13:57:58.875455 1545 sys_catalog.cc:458] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [sys.catalog]: This master's current role is: LEADER
I20260927 13:57:58.875456 1546 sys_catalog.cc:458] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e [sys.catalog]: This master's current role is: LEADER
I20260927 13:57:58.876327 1551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:57:58.893720 1551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:57:58.894836 1539 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 13:57:58.911685 1551 catalog_manager.cc:1347] Generated new cluster ID: c765a2d4222a491e8a0c879e763c42dd
I20260927 13:57:58.911849 1551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:57:58.946988 1551 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:57:58.949023 1551 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:57:58.962730 1551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e: Generated new TSK 0
I20260927 13:57:58.963853 1551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:57:59.756645 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.26.212:54915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35121
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:58:00 dist-test-slave-gkks krb5kdc[1027](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790517480, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 13:58:00.214089 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:00.214468 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:00.214682 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:00.226186 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:00.226326 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:00.226568 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:58:00.241984 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.26.212:54915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:00.243789 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:00.245580 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:00.272473 1594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:00.273088 1595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:00.276314 1584 server_base.cc:1034] running on GCE node
W20260927 13:58:00.276908 1598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:00.277753 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:00.279121 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:00.280427 1584 hybrid_clock.cc:648] HybridClock initialized: now 1790517480280319 us; error 92 us; skew 500 ppm
I20260927 13:58:00.280825 1584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:00 dist-test-slave-gkks krb5kdc[1027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517480, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:00.285168 1584 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:58:00.292724 1584 webserver.cc:460] Webserver started at http://127.22.26.193:44235/ using document root <none> and password file <none>
I20260927 13:58:00.293740 1584 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:00.293908 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:58:00.294274 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:58:00 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517480, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:00.316174 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "e1cb35f6d2224c9ab129ca39b0368b76"
format_stamp: "Formatted at 2026-09-27 13:58:00 on dist-test-slave-gkks"
server_key: "0838d047ec5bf080eeef9c1d9a3585aa"
server_key_iv: "b72e2f60a27218de6b87d1e075499d7f"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:00.317454 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "e1cb35f6d2224c9ab129ca39b0368b76"
format_stamp: "Formatted at 2026-09-27 13:58:00 on dist-test-slave-gkks"
server_key: "0838d047ec5bf080eeef9c1d9a3585aa"
server_key_iv: "b72e2f60a27218de6b87d1e075499d7f"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:00.333587 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.009s sys 0.004s
I20260927 13:58:00.360214 1604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:00.361980 1584 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.008s sys 0.000s
I20260927 13:58:00.362159 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "e1cb35f6d2224c9ab129ca39b0368b76"
format_stamp: "Formatted at 2026-09-27 13:58:00 on dist-test-slave-gkks"
server_key: "0838d047ec5bf080eeef9c1d9a3585aa"
server_key_iv: "b72e2f60a27218de6b87d1e075499d7f"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:00.362411 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:58:00.389825 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:00.394984 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:00.395414 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:00.407586 1584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:58:00.409898 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:58:00.410060 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:00.410266 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:58:00.410406 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:00.551231 1584 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:46149
I20260927 13:58:00.551281 1716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:46149 every 8 connection(s)
I20260927 13:58:00.553027 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:58:00.561378 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 1584
I20260927 13:58:00.561589 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:58:00.562444 22635 external_mini_cluster.cc:1428] Setting key 2212fa6dc671daaac4c5b637b01faf80
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
Sep 27 13:58:00 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517480, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:58:00.612123 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:00.556448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:55139 (local address 127.22.26.254:35121)
0927 13:58:00.576580 (+ 20132us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:00.576588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:00.576613 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:58:00.582428 (+ 5815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:00.582438 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:00.582537 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:00.582629 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:00.594337 (+ 11708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:00.596412 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:00.599414 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:00.600057 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:00.600168 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:00.604706 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:00.604715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:00.604720 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:00.604761 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:00.608448 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:00.609337 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:00.609357 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:00.609361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:00.609524 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:00.610033 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:00.610038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:00.610041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:00.610405 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:00.610551 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:00.611471 (+ 920us) server_negotiation.cc:299] Negotiation successful
0927 13:58:00.611713 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19993,"thread_start_us":20138,"threads_started":1}
Sep 27 13:58:00 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517480, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:00.613322 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:00.557067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35121 (local address 127.22.26.193:55139)
0927 13:58:00.574538 (+ 17471us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:00.574609 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:00.582069 (+ 7460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:00.586436 (+ 4367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:00.586452 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:00.587179 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:00.588139 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:00.588172 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:00.596933 (+ 8761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:00.596938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:00.599213 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:00.599230 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:00.600542 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:00.600549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:00.600852 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:00.601612 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:00.601644 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:00.604427 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:00.608602 (+ 4175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:00.608611 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:00.608631 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:00.609207 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:00.609660 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:00.609665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:00.609669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:00.609922 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:00.610559 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:00.610567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:00.610993 (+ 426us) client_negotiation.cc:769] Sending connection context
0927 13:58:00.611346 (+ 353us) client_negotiation.cc:240] Negotiation successful
0927 13:58:00.611885 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17108,"thread_start_us":23771,"threads_started":1}
I20260927 13:58:00.615469 1717 heartbeater.cc:347] Connected to a master server at 127.22.26.254:35121
I20260927 13:58:00.615809 1717 heartbeater.cc:464] Registering TS with master...
I20260927 13:58:00.616617 1717 heartbeater.cc:511] Master 127.22.26.254:35121 requested a full tablet report, sending...
I20260927 13:58:00.619026 1504 ts_manager.cc:194] Registered new tserver with Master: e1cb35f6d2224c9ab129ca39b0368b76 (127.22.26.193:46149)
I20260927 13:58:00.620880 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:55139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:58:00.643287 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.26.212:54915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35121
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:58:00 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517480, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
W20260927 13:58:01.095733 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:01.096143 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:01.096398 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:01.107991 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:01.108192 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:01.108400 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:58:01.123811 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.26.212:54915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:01.125578 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:01.127341 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:01.142062 1738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:01.142314 1740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:01.142796 1737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:01.143939 1723 server_base.cc:1034] running on GCE node
I20260927 13:58:01.144598 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:01.145529 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:01.146759 1723 hybrid_clock.cc:648] HybridClock initialized: now 1790517481146707 us; error 42 us; skew 500 ppm
I20260927 13:58:01.147125 1723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:01 dist-test-slave-gkks krb5kdc[1027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517481, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:01.150254 1723 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:58:01.151785 1723 webserver.cc:460] Webserver started at http://127.22.26.194:35099/ using document root <none> and password file <none>
I20260927 13:58:01.152456 1723 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:01.152585 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:58:01.152897 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:58:01 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517481, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:01.169955 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "0c86151f411f4af4a4ebc55ccc4d56a9"
format_stamp: "Formatted at 2026-09-27 13:58:01 on dist-test-slave-gkks"
server_key: "9a855f8b0d7f887b9a0e92509b5d4b84"
server_key_iv: "00d90d80b80240b0d855faa3222c65e6"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:01.171007 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "0c86151f411f4af4a4ebc55ccc4d56a9"
format_stamp: "Formatted at 2026-09-27 13:58:01 on dist-test-slave-gkks"
server_key: "9a855f8b0d7f887b9a0e92509b5d4b84"
server_key_iv: "00d90d80b80240b0d855faa3222c65e6"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:01.177800 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 13:58:01.193084 1747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:01.194967 1723 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 13:58:01.195171 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "0c86151f411f4af4a4ebc55ccc4d56a9"
format_stamp: "Formatted at 2026-09-27 13:58:01 on dist-test-slave-gkks"
server_key: "9a855f8b0d7f887b9a0e92509b5d4b84"
server_key_iv: "00d90d80b80240b0d855faa3222c65e6"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:01.195431 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:58:01.223788 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:01.228068 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:01.228384 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:01.229384 1723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:58:01.230931 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:58:01.231031 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:01.231206 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:58:01.231281 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:01.274627 1723 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:40253
I20260927 13:58:01.274644 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:40253 every 8 connection(s)
I20260927 13:58:01.276006 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:58:01.277136 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 1723
I20260927 13:58:01.277309 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:58:01.277887 22635 external_mini_cluster.cc:1428] Setting key b0af75a12755a251b024b87ab17761ae
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Sep 27 13:58:01 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517481, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:58:01.325373 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:01.279342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:48819 (local address 127.22.26.254:35121)
0927 13:58:01.306055 (+ 26713us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:01.306061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:01.306082 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:58:01.306146 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:01.306150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:01.306243 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:01.306351 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:01.308783 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.309891 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.312870 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.313431 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.313525 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:01.319438 (+ 5913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:01.319444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:01.319447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:01.319481 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:01.322156 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:01.322917 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:01.322930 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:01.322933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:01.323057 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:01.323571 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:01.323577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:01.323580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:01.323871 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:01.324028 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:01.324749 (+ 721us) server_negotiation.cc:299] Negotiation successful
0927 13:58:01.324986 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26574,"thread_start_us":27105,"threads_started":1}
I20260927 13:58:01.326542 1863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:01.280325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35121 (local address 127.22.26.194:48819)
0927 13:58:01.302244 (+ 21919us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:01.302305 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:01.304902 (+ 2597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:01.306898 (+ 1996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:01.306911 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:01.307666 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:01.308561 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:01.308588 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.310079 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.310084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:01.312109 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:01.312125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.313875 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.313882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:01.314170 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:01.315050 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:01.315108 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:01.319072 (+ 3964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:01.322310 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:01.322320 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:01.322337 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:01.322775 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:01.323219 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:01.323223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:01.323227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:01.323447 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:01.324003 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:01.324011 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:01.324375 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 13:58:01.324646 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 13:58:01.325216 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20596,"thread_start_us":20699,"threads_started":1}
I20260927 13:58:01.328287 1860 heartbeater.cc:347] Connected to a master server at 127.22.26.254:35121
I20260927 13:58:01.328608 1860 heartbeater.cc:464] Registering TS with master...
I20260927 13:58:01.329456 1860 heartbeater.cc:511] Master 127.22.26.254:35121 requested a full tablet report, sending...
I20260927 13:58:01.331257 1504 ts_manager.cc:194] Registered new tserver with Master: 0c86151f411f4af4a4ebc55ccc4d56a9 (127.22.26.194:40253)
I20260927 13:58:01.332329 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:48819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:58:01.350474 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.26.212:54915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35121
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:58:01.624289 1717 heartbeater.cc:503] Master 127.22.26.254:35121 was elected leader, sending a full tablet report...
W20260927 13:58:01.683954 1866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:01.684208 1866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:01.684300 1866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:01.691624 1866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:01.691720 1866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:01.691844 1866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:58:01.701308 1866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.26.212:54915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:01.702476 1866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:01.703766 1866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:01.721071 1874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:01.721185 1872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:01.721415 1871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:01.722191 1866 server_base.cc:1034] running on GCE node
I20260927 13:58:01.722708 1866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:01.723630 1866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:01.724875 1866 hybrid_clock.cc:648] HybridClock initialized: now 1790517481724820 us; error 58 us; skew 500 ppm
I20260927 13:58:01.725225 1866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:01 dist-test-slave-gkks krb5kdc[1027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517481, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:01.728255 1866 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:58:01.729668 1866 webserver.cc:460] Webserver started at http://127.22.26.195:39371/ using document root <none> and password file <none>
I20260927 13:58:01.730320 1866 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:01.730464 1866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:58:01.730774 1866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:58:01 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517481, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:01.746869 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "969b9b5a999d480780d054c1462c004b"
format_stamp: "Formatted at 2026-09-27 13:58:01 on dist-test-slave-gkks"
server_key: "8c9e0e2e7d5046a825ad08c9defb8a21"
server_key_iv: "16ca1580f204d6f5fe91531d09b51925"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:01.747954 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "969b9b5a999d480780d054c1462c004b"
format_stamp: "Formatted at 2026-09-27 13:58:01 on dist-test-slave-gkks"
server_key: "8c9e0e2e7d5046a825ad08c9defb8a21"
server_key_iv: "16ca1580f204d6f5fe91531d09b51925"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:01.754640 1866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 13:58:01.768491 1881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:01.770000 1866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:58:01.770182 1866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "969b9b5a999d480780d054c1462c004b"
format_stamp: "Formatted at 2026-09-27 13:58:01 on dist-test-slave-gkks"
server_key: "8c9e0e2e7d5046a825ad08c9defb8a21"
server_key_iv: "16ca1580f204d6f5fe91531d09b51925"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:01.770401 1866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:58:01.799052 1866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:01.803272 1866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:01.803598 1866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:01.804533 1866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:58:01.805877 1866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:58:01.805951 1866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:01.806120 1866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:58:01.806171 1866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:01.842626 1866 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:42711
I20260927 13:58:01.842661 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:42711 every 8 connection(s)
I20260927 13:58:01.843976 1866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:58:01.846185 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 1866
I20260927 13:58:01.846354 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:58:01.847124 22635 external_mini_cluster.cc:1428] Setting key a6b42404577a6c820f8722e3f4d1a00b
Sep 27 13:58:01 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517481, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:58:01.864753 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:01.846850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:39577 (local address 127.22.26.254:35121)
0927 13:58:01.847393 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:01.847399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:01.848671 (+ 1272us) server_negotiation.cc:407] Connection header received
0927 13:58:01.850945 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:01.850950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:01.851036 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:01.851193 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:01.852562 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.853280 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.854821 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.855313 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.855453 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:01.858656 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:01.858661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:01.858664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:01.858695 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:01.861815 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:01.862707 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:01.862710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:01.862713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:01.862875 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:01.863350 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:01.863353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:01.863355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:01.863627 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:01.863732 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:01.864333 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 13:58:01.864508 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":245,"threads_started":1}
I20260927 13:58:01.865713 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:01.847555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35121 (local address 127.22.26.195:39577)
0927 13:58:01.848499 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:01.848554 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:01.850673 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:01.851421 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:01.851430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:01.851849 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:01.852406 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:01.852423 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.853436 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.853441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:01.854668 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:01.854680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.855533 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.855540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:01.855787 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:01.856412 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:01.856435 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:01.858451 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:01.861972 (+ 3521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:01.861980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:01.861995 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:01.862589 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:01.862995 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:01.862998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:01.863000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:01.863256 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:01.863739 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:01.863744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:01.864035 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 13:58:01.864254 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 13:58:01.864598 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":364,"threads_started":1}
I20260927 13:58:01.867039 1994 heartbeater.cc:347] Connected to a master server at 127.22.26.254:35121
I20260927 13:58:01.867441 1994 heartbeater.cc:464] Registering TS with master...
I20260927 13:58:01.868116 1994 heartbeater.cc:511] Master 127.22.26.254:35121 requested a full tablet report, sending...
I20260927 13:58:01.869424 1504 ts_manager.cc:194] Registered new tserver with Master: 969b9b5a999d480780d054c1462c004b (127.22.26.195:42711)
I20260927 13:58:01.870157 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:39577
I20260927 13:58:01.874948 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:58:01.892162 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:01.879699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.22.26.254:35121)
0927 13:58:01.879867 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:01.879871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:01.880412 (+ 541us) server_negotiation.cc:407] Connection header received
0927 13:58:01.880543 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:01.880546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:01.880613 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:01.880700 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:01.882089 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.882853 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.884231 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.884631 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.884729 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:01.886951 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:01.886957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:01.886963 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:01.887012 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:01.889776 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:01.890432 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:01.890436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:01.890439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:01.890559 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:01.890919 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:01.890923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:01.890925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:01.891287 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:01.891499 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:01.891757 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 13:58:01.891938 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:58:01.896076 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:58:01.903417 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:58:01.941344 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:01.916255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:42711 (local address 127.0.0.1:41228)
0927 13:58:01.928227 (+ 11972us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:01.928302 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:01.928495 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:01.930776 (+ 2281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:01.930784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:01.930821 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:58:01.931354 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:01.931369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.934195 (+ 2826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.934204 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:01.937308 (+ 3104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:01.937331 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.939939 (+ 2608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.939949 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:01.940266 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:01.940341 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 13:58:01.940509 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 13:58:01.940792 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11758,"thread_start_us":11636,"threads_started":1}
I20260927 13:58:01.941800 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:01.916926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.22.26.195:42711)
0927 13:58:01.929806 (+ 12880us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:01.929818 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:01.929869 (+ 51us) server_negotiation.cc:407] Connection header received
0927 13:58:01.929985 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:01.929996 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:01.930298 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:01.931606 (+ 1308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:58:01.931954 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.933996 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.938017 (+ 4021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.939625 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.940932 (+ 1307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:01.941190 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:01.941283 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 13:58:01.941456 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12654,"thread_start_us":13524,"threads_started":1}
I20260927 13:58:01.941977 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:01.916387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.22.26.193:46149)
0927 13:58:01.927725 (+ 11338us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:01.927739 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:01.928906 (+ 1167us) server_negotiation.cc:407] Connection header received
0927 13:58:01.929047 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:01.929060 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:01.929396 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:01.930593 (+ 1197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:58:01.930881 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.933028 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.937576 (+ 4548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.939698 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.939879 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:01.940214 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:01.941439 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0927 13:58:01.941635 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11089,"thread_start_us":10868,"threads_started":1}
I20260927 13:58:01.942457 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:01.916075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:46149 (local address 127.0.0.1:55320)
0927 13:58:01.928227 (+ 12152us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:01.928678 (+ 451us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:01.928786 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:01.929690 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:01.929701 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:01.929751 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:58:01.930327 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:01.930345 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.933226 (+ 2881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.933235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:01.936411 (+ 3176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:01.936431 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.940692 (+ 4261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.940701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:01.940903 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:01.940935 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 13:58:01.941022 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 13:58:01.941172 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11835,"mutex_wait_us":220,"thread_start_us":12155,"threads_started":1}
I20260927 13:58:01.945886 1929 tablet_service.cc:1467] Processing CreateTablet for tablet b34d774e28774142b2ab69d0c715b4bb (DEFAULT_TABLE table=test-table [id=7d20f5601bcf46508aa5cc6379589f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:58:01.946278 1652 tablet_service.cc:1467] Processing CreateTablet for tablet b34d774e28774142b2ab69d0c715b4bb (DEFAULT_TABLE table=test-table [id=7d20f5601bcf46508aa5cc6379589f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:58:01.947443 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34d774e28774142b2ab69d0c715b4bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:58:01.947726 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34d774e28774142b2ab69d0c715b4bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:58:01.951311 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:01.928879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:40253 (local address 127.0.0.1:37656)
0927 13:58:01.940629 (+ 11750us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:01.940649 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:01.940768 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:01.944017 (+ 3249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:01.944021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:01.944048 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:58:01.944428 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:01.944441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.946260 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.946265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:01.948667 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:01.948681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.950498 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.950504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:01.950663 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:01.950701 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 13:58:01.950804 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 13:58:01.950940 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11603,"thread_start_us":11445,"threads_started":1}
I20260927 13:58:01.951579 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:01.929177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.22.26.194:40253)
0927 13:58:01.943391 (+ 14214us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:01.943399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:01.943429 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:58:01.943531 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:01.943538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:01.943805 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:01.944023 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:58:01.944611 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.946107 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.948840 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:01.950274 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:01.950415 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:01.950593 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:01.950864 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 13:58:01.951033 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14029,"thread_start_us":14389,"threads_started":1}
I20260927 13:58:01.955296 1795 tablet_service.cc:1467] Processing CreateTablet for tablet b34d774e28774142b2ab69d0c715b4bb (DEFAULT_TABLE table=test-table [id=7d20f5601bcf46508aa5cc6379589f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:58:01.956607 1795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34d774e28774142b2ab69d0c715b4bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:58:01.959630 2013 tablet_bootstrap.cc:492] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76: Bootstrap starting.
I20260927 13:58:01.961639 2014 tablet_bootstrap.cc:492] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b: Bootstrap starting.
I20260927 13:58:01.963389 2013 tablet_bootstrap.cc:654] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76: Neither blocks nor log segments found. Creating new log.
I20260927 13:58:01.964709 2013 log.cc:824] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76: Log is configured to *not* fsync() on all Append() calls
I20260927 13:58:01.965085 2014 tablet_bootstrap.cc:654] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b: Neither blocks nor log segments found. Creating new log.
I20260927 13:58:01.966594 2014 log.cc:824] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:58:01.968503 2013 tablet_bootstrap.cc:492] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76: No bootstrap required, opened a new log
I20260927 13:58:01.968771 2013 ts_tablet_manager.cc:1397] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 13:58:01.969291 2014 tablet_bootstrap.cc:492] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b: No bootstrap required, opened a new log
I20260927 13:58:01.969594 2014 ts_tablet_manager.cc:1397] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 13:58:01.969763 2017 tablet_bootstrap.cc:492] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9: Bootstrap starting.
I20260927 13:58:01.973323 2017 tablet_bootstrap.cc:654] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9: Neither blocks nor log segments found. Creating new log.
I20260927 13:58:01.974553 2017 log.cc:824] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9: Log is configured to *not* fsync() on all Append() calls
I20260927 13:58:01.977748 2013 raft_consensus.cc:348] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46149 } } peers { permanent_uuid: "969b9b5a999d480780d054c1462c004b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42711 } } peers { permanent_uuid: "0c86151f411f4af4a4ebc55ccc4d56a9" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40253 } }
I20260927 13:58:01.978108 2013 raft_consensus.cc:374] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:58:01.978211 2013 raft_consensus.cc:729] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cb35f6d2224c9ab129ca39b0368b76, State: Initialized, Role: FOLLOWER
I20260927 13:58:01.978809 2013 consensus_queue.cc:260] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46149 } } peers { permanent_uuid: "969b9b5a999d480780d054c1462c004b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42711 } } peers { permanent_uuid: "0c86151f411f4af4a4ebc55ccc4d56a9" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40253 } }
I20260927 13:58:01.980365 2013 ts_tablet_manager.cc:1428] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 13:58:01.981352 2014 raft_consensus.cc:348] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46149 } } peers { permanent_uuid: "969b9b5a999d480780d054c1462c004b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42711 } } peers { permanent_uuid: "0c86151f411f4af4a4ebc55ccc4d56a9" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40253 } }
I20260927 13:58:01.981750 2014 raft_consensus.cc:374] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:58:01.981873 2014 raft_consensus.cc:729] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969b9b5a999d480780d054c1462c004b, State: Initialized, Role: FOLLOWER
I20260927 13:58:01.982489 2014 consensus_queue.cc:260] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46149 } } peers { permanent_uuid: "969b9b5a999d480780d054c1462c004b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42711 } } peers { permanent_uuid: "0c86151f411f4af4a4ebc55ccc4d56a9" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40253 } }
I20260927 13:58:01.983590 1994 heartbeater.cc:503] Master 127.22.26.254:35121 was elected leader, sending a full tablet report...
I20260927 13:58:01.984243 2014 ts_tablet_manager.cc:1428] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 13:58:01.984506 2017 tablet_bootstrap.cc:492] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9: No bootstrap required, opened a new log
I20260927 13:58:01.984798 2017 ts_tablet_manager.cc:1397] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260927 13:58:01.992403 2017 raft_consensus.cc:348] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46149 } } peers { permanent_uuid: "969b9b5a999d480780d054c1462c004b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42711 } } peers { permanent_uuid: "0c86151f411f4af4a4ebc55ccc4d56a9" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40253 } }
I20260927 13:58:01.992702 2017 raft_consensus.cc:374] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:58:01.992767 2017 raft_consensus.cc:729] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c86151f411f4af4a4ebc55ccc4d56a9, State: Initialized, Role: FOLLOWER
I20260927 13:58:01.993263 2017 consensus_queue.cc:260] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46149 } } peers { permanent_uuid: "969b9b5a999d480780d054c1462c004b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42711 } } peers { permanent_uuid: "0c86151f411f4af4a4ebc55ccc4d56a9" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40253 } }
I20260927 13:58:01.994140 1860 heartbeater.cc:503] Master 127.22.26.254:35121 was elected leader, sending a full tablet report...
I20260927 13:58:01.994817 2017 ts_tablet_manager.cc:1428] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260927 13:58:02.026211 2019 raft_consensus.cc:482] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:58:02.026537 2019 raft_consensus.cc:504] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46149 } } peers { permanent_uuid: "969b9b5a999d480780d054c1462c004b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42711 } } peers { permanent_uuid: "0c86151f411f4af4a4ebc55ccc4d56a9" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40253 } }
I20260927 13:58:02.028296 2019 leader_election.cc:288] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 969b9b5a999d480780d054c1462c004b (127.22.26.195:42711), 0c86151f411f4af4a4ebc55ccc4d56a9 (127.22.26.194:40253)
I20260927 13:58:02.035343 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:02.028988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:34777 (local address 127.22.26.194:40253)
0927 13:58:02.029180 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:02.029190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:02.029627 (+ 437us) server_negotiation.cc:407] Connection header received
0927 13:58:02.029817 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:02.029821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:02.029880 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:02.029986 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:58:02.030547 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.031329 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.033272 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.034529 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.034682 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:02.034753 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:02.034992 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 13:58:02.035142 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 13:58:02.035557 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:02.028599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:40253 (local address 127.22.26.193:34777)
0927 13:58:02.029511 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:02.029527 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:02.029657 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:02.030000 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:02.030005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:02.030028 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:58:02.030404 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:02.030414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.031447 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.031454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:02.033128 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:02.033139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.034711 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.034715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:02.034825 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:02.034849 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 13:58:02.034957 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 13:58:02.035139 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":3,"thread_start_us":706,"threads_started":1}
I20260927 13:58:02.035787 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:02.028716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:39625 (local address 127.22.26.195:42711)
0927 13:58:02.028948 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:02.028957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:02.029737 (+ 780us) server_negotiation.cc:407] Connection header received
0927 13:58:02.029934 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:02.029939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:02.030017 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:02.030114 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:58:02.030740 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.031543 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.033620 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.034856 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.035041 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:02.035133 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:02.035439 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 13:58:02.035566 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:58:02.035992 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:02.028743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:42711 (local address 127.22.26.193:39625)
0927 13:58:02.029620 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:02.029639 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:02.029796 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:02.030131 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:02.030136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:02.030173 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:58:02.030589 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:02.030598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.031674 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.031679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:02.033405 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:02.033438 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.035104 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.035110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:02.035269 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:02.035303 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 13:58:02.035395 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 13:58:02.035542 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":20,"thread_start_us":400,"threads_started":1}
I20260927 13:58:02.036701 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34d774e28774142b2ab69d0c715b4bb" candidate_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c86151f411f4af4a4ebc55ccc4d56a9" is_pre_election: true
I20260927 13:58:02.036999 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34d774e28774142b2ab69d0c715b4bb" candidate_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969b9b5a999d480780d054c1462c004b" is_pre_election: true
I20260927 13:58:02.037170 1815 raft_consensus.cc:2454] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1cb35f6d2224c9ab129ca39b0368b76 in term 0.
I20260927 13:58:02.037490 1949 raft_consensus.cc:2454] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1cb35f6d2224c9ab129ca39b0368b76 in term 0.
I20260927 13:58:02.037902 1605 leader_election.cc:302] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c86151f411f4af4a4ebc55ccc4d56a9, e1cb35f6d2224c9ab129ca39b0368b76; no voters:
I20260927 13:58:02.038328 2019 raft_consensus.cc:2781] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:58:02.038492 2019 raft_consensus.cc:482] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:58:02.038604 2019 raft_consensus.cc:3037] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:58:02.040769 2019 raft_consensus.cc:504] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46149 } } peers { permanent_uuid: "969b9b5a999d480780d054c1462c004b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42711 } } peers { permanent_uuid: "0c86151f411f4af4a4ebc55ccc4d56a9" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40253 } }
I20260927 13:58:02.041316 2019 leader_election.cc:288] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [CANDIDATE]: Term 1 election: Requested vote from peers 969b9b5a999d480780d054c1462c004b (127.22.26.195:42711), 0c86151f411f4af4a4ebc55ccc4d56a9 (127.22.26.194:40253)
I20260927 13:58:02.041779 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34d774e28774142b2ab69d0c715b4bb" candidate_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969b9b5a999d480780d054c1462c004b"
I20260927 13:58:02.041782 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34d774e28774142b2ab69d0c715b4bb" candidate_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c86151f411f4af4a4ebc55ccc4d56a9"
I20260927 13:58:02.042002 1949 raft_consensus.cc:3037] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:58:02.041990 1815 raft_consensus.cc:3037] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:58:02.046202 1815 raft_consensus.cc:2454] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1cb35f6d2224c9ab129ca39b0368b76 in term 1.
I20260927 13:58:02.046202 1949 raft_consensus.cc:2454] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1cb35f6d2224c9ab129ca39b0368b76 in term 1.
I20260927 13:58:02.046693 1605 leader_election.cc:302] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c86151f411f4af4a4ebc55ccc4d56a9, e1cb35f6d2224c9ab129ca39b0368b76; no voters:
I20260927 13:58:02.047166 2019 raft_consensus.cc:2781] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:58:02.047928 2019 raft_consensus.cc:686] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [term 1 LEADER]: Becoming Leader. State: Replica: e1cb35f6d2224c9ab129ca39b0368b76, State: Running, Role: LEADER
I20260927 13:58:02.048725 2019 consensus_queue.cc:237] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46149 } } peers { permanent_uuid: "969b9b5a999d480780d054c1462c004b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42711 } } peers { permanent_uuid: "0c86151f411f4af4a4ebc55ccc4d56a9" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40253 } }
I20260927 13:58:02.054280 1503 catalog_manager.cc:5495] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1cb35f6d2224c9ab129ca39b0368b76 (127.22.26.193). New cstate: current_term: 1 leader_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cb35f6d2224c9ab129ca39b0368b76" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "969b9b5a999d480780d054c1462c004b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c86151f411f4af4a4ebc55ccc4d56a9" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40253 } health_report { overall_health: UNKNOWN } } }
W20260927 13:58:02.062069 1718 tablet.cc:2369] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:58:02.063340 1861 tablet.cc:2369] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:58:02.095571 1995 tablet.cc:2369] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:58:02.101545 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:02.096261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.22.26.193:46149)
0927 13:58:02.096457 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:02.096461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:02.096478 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:58:02.096684 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:02.096688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:02.096764 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:02.096893 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:58:02.097217 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.097938 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.099436 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.099828 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.099929 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:02.100461 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:58:02.100951 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:58:02.101110 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:02.101189 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 13:58:02.101265 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 13:58:02.109277 1815 raft_consensus.cc:1261] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9 [term 1 FOLLOWER]: Refusing update from remote peer e1cb35f6d2224c9ab129ca39b0368b76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:58:02.109274 1949 raft_consensus.cc:1261] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b [term 1 FOLLOWER]: Refusing update from remote peer e1cb35f6d2224c9ab129ca39b0368b76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:58:02.110234 2019 consensus_queue.cc:1035] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "969b9b5a999d480780d054c1462c004b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:58:02.110527 2024 consensus_queue.cc:1035] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c86151f411f4af4a4ebc55ccc4d56a9" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:58:02.126286 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333959606709977088 to 7333959606475993088. Current Snapshot: MvccSnapshot[applied={T|T < 7333959606709977088}]
I20260927 13:58:02.129006 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333959606709977088 to 7333959606475993088. Current Snapshot: MvccSnapshot[applied={T|T < 7333959606709977088}]
I20260927 13:58:02.138963 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:58:02.139298 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:58:02.142858 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e605987c74f14d2ab879e9912335e26e: Sending DeleteTablet for 3 replicas of tablet b34d774e28774142b2ab69d0c715b4bb
I20260927 13:58:02.144002 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet b34d774e28774142b2ab69d0c715b4bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:58:02 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:41228
I20260927 13:58:02.144039 1652 tablet_service.cc:1514] Processing DeleteTablet for tablet b34d774e28774142b2ab69d0c715b4bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:58:02 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:55320
I20260927 13:58:02.144183 1795 tablet_service.cc:1514] Processing DeleteTablet for tablet b34d774e28774142b2ab69d0c715b4bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:58:02 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:37656
I20260927 13:58:02.147017 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 1584
I20260927 13:58:02.176854 2037 tablet_replica.cc:330] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b: stopping tablet replica
I20260927 13:58:02.177604 2037 raft_consensus.cc:2229] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:58:02.182906 2037 raft_consensus.cc:2258] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:58:02.185935 2037 ts_tablet_manager.cc:1905] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:58:02.190919 2036 tablet_replica.cc:330] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76: stopping tablet replica
I20260927 13:58:02.191642 2036 raft_consensus.cc:2229] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:58:02.191968 2038 tablet_replica.cc:330] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9: stopping tablet replica
I20260927 13:58:02.192548 2036 raft_consensus.cc:2258] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 13:58:02 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:58:02.194977 2036 ts_tablet_manager.cc:1905] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:58:02.196933 2038 raft_consensus.cc:2229] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:58:02.197518 2038 raft_consensus.cc:2258] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:58:02.198843 2037 ts_tablet_manager.cc:1918] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:58:02.199095 2037 log.cc:1196] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/b34d774e28774142b2ab69d0c715b4bb
I20260927 13:58:02.199695 2037 ts_tablet_manager.cc:1939] T b34d774e28774142b2ab69d0c715b4bb P 969b9b5a999d480780d054c1462c004b: Deleting consensus metadata
I20260927 13:58:02.201697 1490 catalog_manager.cc:4841] TS 969b9b5a999d480780d054c1462c004b (127.22.26.195:42711): tablet b34d774e28774142b2ab69d0c715b4bb (table test-table [id=7d20f5601bcf46508aa5cc6379589f5a]) successfully deleted
I20260927 13:58:02.202589 2038 ts_tablet_manager.cc:1905] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:58:02.204497 2036 ts_tablet_manager.cc:1918] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:58:02.204703 2036 log.cc:1196] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/b34d774e28774142b2ab69d0c715b4bb
I20260927 13:58:02.205171 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:02.148805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.22.26.193:46149)
0927 13:58:02.149042 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:02.149051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:02.173070 (+ 24019us) server_negotiation.cc:407] Connection header received
0927 13:58:02.173394 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:02.173401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:02.173522 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:02.173619 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:02.178082 (+ 4463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.179788 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.186257 (+ 6469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.187394 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.187536 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:02.195874 (+ 8338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:02.195891 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:02.195914 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:02.195966 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:02.200110 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:02.200987 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:02.200994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:02.200998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:02.201158 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:02.202321 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:02.202328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:02.202331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:02.202798 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:02.203153 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:02.204363 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0927 13:58:02.204606 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 13:58:02.205294 2036 ts_tablet_manager.cc:1939] T b34d774e28774142b2ab69d0c715b4bb P e1cb35f6d2224c9ab129ca39b0368b76: Deleting consensus metadata
I20260927 13:58:02.206018 1712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:58:02.209978 2038 ts_tablet_manager.cc:1918] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:58:02.210201 2038 log.cc:1196] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/b34d774e28774142b2ab69d0c715b4bb
I20260927 13:58:02.210773 2038 ts_tablet_manager.cc:1939] T b34d774e28774142b2ab69d0c715b4bb P 0c86151f411f4af4a4ebc55ccc4d56a9: Deleting consensus metadata
I20260927 13:58:02.212263 1488 catalog_manager.cc:4841] TS 0c86151f411f4af4a4ebc55ccc4d56a9 (127.22.26.194:40253): tablet b34d774e28774142b2ab69d0c715b4bb (table test-table [id=7d20f5601bcf46508aa5cc6379589f5a]) successfully deleted
I20260927 13:58:02.307317 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 1584
I20260927 13:58:02.307471 1488 catalog_manager.cc:4841] TS e1cb35f6d2224c9ab129ca39b0368b76 (127.22.26.193:46149): tablet b34d774e28774142b2ab69d0c715b4bb (table test-table [id=7d20f5601bcf46508aa5cc6379589f5a]) successfully deleted
I20260927 13:58:02.326654 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 1723
Sep 27 13:58:02 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:58:02.341243 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:02.327480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.22.26.194:40253)
0927 13:58:02.327647 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:02.327652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:02.327732 (+ 80us) server_negotiation.cc:407] Connection header received
0927 13:58:02.327922 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:02.327925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:02.327989 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:02.328087 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:02.329427 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.330161 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.331563 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.331944 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.332045 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:02.336050 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:02.336062 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:02.336076 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:02.336115 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:02.338813 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:02.339537 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:02.339544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:02.339546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:02.339673 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:02.340015 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:02.340018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:02.340020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:02.340371 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:02.340587 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:02.340833 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 13:58:02.340984 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:58:02.341909 1855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:58:02.430876 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 1723
I20260927 13:58:02.447368 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 1866
Sep 27 13:58:02 dist-test-slave-gkks krb5kdc[1027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:58:02.462536 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:02.448269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41232 (local address 127.22.26.195:42711)
0927 13:58:02.448488 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:02.448496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:02.448516 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:58:02.448696 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:02.448704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:02.448776 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:02.448890 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:02.450266 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.450998 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.452340 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:02.452733 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:02.452863 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:02.457098 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:02.457113 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:02.457128 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:02.457160 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:02.459790 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:02.460472 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:02.460480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:02.460482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:02.460594 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:02.460978 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:02.460984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:02.460988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:02.461348 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:02.461643 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:02.461938 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 13:58:02.462172 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 13:58:02.463249 1989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:58:02.551873 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 1866
I20260927 13:58:02.568171 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 1472
I20260927 13:58:02.569290 1534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:58:02.680317 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 1472
I20260927 13:58:02.701985 22635 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 13:58:03.036768 22635 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 13:58:03.037020 22635 mini_ranger.cc:66] Stopping Ranger...
I20260927 13:58:03.382104 22635 mini_ranger.cc:68] Stopped Ranger
2026-09-27 13:58:03.382 UTC [1057] LOG: received smart shutdown request
2026-09-27 13:58:03.385 UTC [1057] LOG: background worker "logical replication launcher" (PID 1066) exited with exit code 1
2026-09-27 13:58:03.386 UTC [1061] LOG: shutting down
2026-09-27 13:58:03.578 UTC [1057] LOG: database system is shut down
2026-09-27T13:58:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52655 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:58:03 dist-test-slave-gkks krb5kdc[2052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:58:03 dist-test-slave-gkks krb5kdc[2052](info): set up 2 sockets
Sep 27 13:58:03 dist-test-slave-gkks krb5kdc[2052](info): commencing operation
krb5kdc: starting...
W20260927 13:58:04.831964 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:58:04 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:58:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/rangeradmin_127.22.26.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/rangeradmin_127.22.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/rangerlookup_127.22.26.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/rangerlookup_127.22.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/HTTP_127.22.26.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/HTTP_127.22.26.212@KRBTEST.COM.keytab.
I20260927 13:58:05.167970 22635 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUeHuEC/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/postgres -l logfile start
2026-09-27 13:58:07.835 UTC [2082] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 13:58:07.835 UTC [2082] LOG: listening on IPv4 address "127.22.26.212", port 39491
2026-09-27 13:58:07.839 UTC [2082] LOG: listening on Unix socket "/tmp/.s.PGSQL.39491"
2026-09-27 13:58:07.860 UTC [2085] LOG: database system was shut down at 2026-09-27 13:58:06 UTC
2026-09-27 13:58:07.864 UTC [2082] LOG: database system is ready to accept connections
I20260927 13:58:08.839612 22635 mini_postgres.cc:96] Postgres bound to 39491
2026-09-27 13:58:08.857 UTC [2093] FATAL: database "slave" does not exist
127.22.26.212:39491 - accepting connections
I20260927 13:58:08.857923 22635 mini_ranger.cc:161] Starting Ranger...
I20260927 13:58:08.883471 22635 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 13:58:09.428140 22635 mini_ranger.cc:87] Created ranger Postgres database
I20260927 13:58:09.428306 22635 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-admin
2026-09-27 13:58:09,704 [I] DB FLAVOR :POSTGRES
2026-09-27 13:58:09,705 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 13:58:09,705 [I] Checking connection..
2026-09-27 13:58:09,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 13:58:09,963 [I] Checking connection passed.
2026-09-27 13:58:09,963 [I] --------- Verifying version history table ---------
2026-09-27 13:58:09,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:58:10,208 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 13:58:10,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:58:10,439 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 13:58:10,440 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 13:58:10,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 13:58:10.703 UTC [2157] WARNING: there is no transaction in progress
2026-09-27 13:58:10,710 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 13:58:10,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:58:10,949 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 13:58:10,949 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 13:58:10,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 13:58:11,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 13:58:11,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gkks.c.gcp-upstream.internal','N') ;"
2026-09-27 13:58:11,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 13:58:11,908 [I] Table x_portal_user does not exist in database ranger
2026-09-27 13:58:11,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 13:58:12,208 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 13:58:12,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 13:58:12,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 13:58:12,723 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 13:58:12,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 13:58:13.673 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.689 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.706 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.722 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.738 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.757 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.822 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.829 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.844 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.854 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.865 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.878 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.888 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.895 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.902 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:13.910 UTC [2265] WARNING: there is no transaction in progress
2026-09-27 13:58:14,573 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 13:58:14,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 13:58:14,830 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 13:58:14,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 13:58:15,078 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 13:58:15,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 13:58:15,331 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 13:58:15,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 13:58:15,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 13:58:15,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gkks.c.gcp-upstream.internal';"
2026-09-27 13:58:16,086 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 13:58:16,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 13:58:16,341 [I] DB_PATCHES have already been applied
I20260927 13:58:16.347757 22635 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUeHuEC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 13:58:16.347843 22635 mini_ranger.cc:193] Using host: 127.22.26.212
I20260927 13:58:16.362979 22635 mini_ranger.cc:227] Ranger admin URL: http://127.22.26.212:51921
Sep 27, 2026 1:58:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 1:58:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 1:58:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 1:58:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 1:58:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 1:58:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 1:58:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 13:58:17.509804 22635 mini_ranger.cc:160] Time spent starting Ranger: real 8.652s user 0.001s sys 0.038s
Sep 27, 2026 1:58:36 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 1:58:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 1:58:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 1:58:36 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 1:58:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 1:58:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 1:58:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 1:58:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 1:58:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 1:58:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 1:58:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 1:58:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 1:58:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 1:58:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 1:58:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 1:58:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 1:58:36 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 1:58:37 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 13:58:38.134835 22635 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/rangerkms_127.22.26.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/rangerkms_127.22.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.26.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.26.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.26.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/HTTP_127.22.26.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.26.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/HTTP_127.22.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 13:58:38 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517518, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 13:58:38.316468 22635 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 13:58:38.337555 22635 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 13:58:39.308125 22635 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 13:58:39.308257 22635 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-kms
I20260927 13:58:39.308334 22635 mini_ranger_kms.cc:227] Using postgres at 127.22.26.212:39491
2026-09-27 13:58:39,667 [I] DB FLAVOR :POSTGRES
2026-09-27 13:58:39,667 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 13:58:39,667 [I] Checking connection
2026-09-27 13:58:39,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 13:58:39,929 [I] connection success
2026-09-27 13:58:39,929 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 13:58:39,930 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 13:58:39,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 13:58:40,177 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 13:58:40,178 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 13:58:40,178 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 13:58:40,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.26.212:39491/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 13:58:40,436 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 13:58:40.926364 22635 mini_ranger_kms.cc:310] Created kms service
I20260927 13:58:41.114866 22635 mini_ranger_kms.cc:326] Created kudu user
I20260927 13:58:41.163431 22635 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 13:58:41 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517518, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:41.584957 22635 mini_ranger_kms.cc:384] Added ranger policy
I20260927 13:58:41.585366 22635 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUeHuEC/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUeHuEC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 13:58:41.585418 22635 mini_ranger_kms.cc:240] Using host: 127.22.26.212
I20260927 13:58:41.605000 22635 mini_ranger_kms.cc:276] Ranger KMS PID: 2445
I20260927 13:58:41.605175 22635 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.26.212:35947
Sep 27, 2026 1:58:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 1:58:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 1:58:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 1:58:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 1:58:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 1:58:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 13:58:43.990190 22635 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.674s user 0.108s sys 0.265s
I20260927 13:58:43.990502 22635 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 13:58:43.990723 22635 mini_ranger_kms.cc:401] 127.22.26.212:35947/kms/v1/keys
Sep 27 13:58:44 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790517524, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.26.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 13:58:44 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517524, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.26.212@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
Sep 27 13:58:44 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517524, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.26.212@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
Sep 27 13:58:47 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:58:47.159297 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:35929
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:35929
--ranger_config_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.26.212:35947/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:58:47.517407 2496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:47.517668 2496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:47.517807 2496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:47.524839 2496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:58:47.524933 2496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:47.525018 2496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:47.525118 2496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:58:47.525195 2496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:58:47.535059 2496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.26.212:35947/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:35929
--ranger_config_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:35929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:47.536329 2496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:47.537678 2496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:47.551342 2501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:47.551386 2502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:47.551483 2504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:47.552245 2496 server_base.cc:1034] running on GCE node
I20260927 13:58:47.553140 2496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:47.554836 2496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:47.556082 2496 hybrid_clock.cc:648] HybridClock initialized: now 1790517527556029 us; error 49 us; skew 500 ppm
I20260927 13:58:47.556465 2496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:47 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517527, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:47.565248 2496 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:58:47.566906 2496 webserver.cc:460] Webserver started at http://127.22.26.254:34965/ using document root <none> and password file <none>
I20260927 13:58:47.567633 2496 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:47.567761 2496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:58:47.568058 2496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:58:47 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517527, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:47.617800 2496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "749f49e6f8784401a1d39b0b262ceaec"
format_stamp: "Formatted at 2026-09-27 13:58:47 on dist-test-slave-gkks"
server_key: "61687e281edabea2fe2ddaf89b08688d"
server_key_iv: "12cb32d47c96ba27d9b95edccf77c62f"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:47.618700 2496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "749f49e6f8784401a1d39b0b262ceaec"
format_stamp: "Formatted at 2026-09-27 13:58:47 on dist-test-slave-gkks"
server_key: "61687e281edabea2fe2ddaf89b08688d"
server_key_iv: "12cb32d47c96ba27d9b95edccf77c62f"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:47.624271 2496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 13:58:47.640882 2511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:47.642407 2496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:58:47.642591 2496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "749f49e6f8784401a1d39b0b262ceaec"
format_stamp: "Formatted at 2026-09-27 13:58:47 on dist-test-slave-gkks"
server_key: "61687e281edabea2fe2ddaf89b08688d"
server_key_iv: "12cb32d47c96ba27d9b95edccf77c62f"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:47.642809 2496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:58:47.677013 2496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:47.681165 2496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:47.681479 2496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:47.700706 2496 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:35929
I20260927 13:58:47.700745 2562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:35929 every 8 connection(s)
I20260927 13:58:47.702059 2496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:58:47.705736 2563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:58:47.706398 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 2496
I20260927 13:58:47.706606 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:58:47.707401 22635 external_mini_cluster.cc:1428] Setting key 4b42540234f09488d407f0d2b12242a7
I20260927 13:58:47.720113 2563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec: Bootstrap starting.
Sep 27 13:58:47 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:58:47.723758 2563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec: Neither blocks nor log segments found. Creating new log.
I20260927 13:58:47.724808 2563 log.cc:824] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec: Log is configured to *not* fsync() on all Append() calls
I20260927 13:58:47.728143 2563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec: No bootstrap required, opened a new log
I20260927 13:58:47.731902 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:47.709367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.22.26.254:35929)
0927 13:58:47.710194 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:47.710206 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:47.710249 (+ 43us) server_negotiation.cc:407] Connection header received
0927 13:58:47.711901 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:47.711923 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:47.712328 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:47.712681 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:47.714482 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:47.715948 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:47.717981 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:47.718655 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:47.718793 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:47.723480 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:47.723488 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:47.723501 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:47.723543 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:47.727191 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:47.727884 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:47.727893 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:47.727900 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:47.728139 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:47.728604 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:47.728612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:47.728616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:47.729321 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:47.729622 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:47.730023 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 13:58:47.730462 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":392,"threads_started":1}
I20260927 13:58:47.737339 2563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f49e6f8784401a1d39b0b262ceaec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35929 } }
I20260927 13:58:47.737700 2563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:58:47.737846 2563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749f49e6f8784401a1d39b0b262ceaec, State: Initialized, Role: FOLLOWER
I20260927 13:58:47.738476 2563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f49e6f8784401a1d39b0b262ceaec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35929 } }
I20260927 13:58:47.738713 2563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:58:47.738839 2563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:58:47.739006 2563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:58:47.741114 2563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f49e6f8784401a1d39b0b262ceaec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35929 } }
I20260927 13:58:47.741598 2563 leader_election.cc:302] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 749f49e6f8784401a1d39b0b262ceaec; no voters:
I20260927 13:58:47.742146 2563 leader_election.cc:288] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:58:47.742298 2568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:58:47.743664 2568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 1 LEADER]: Becoming Leader. State: Replica: 749f49e6f8784401a1d39b0b262ceaec, State: Running, Role: LEADER
I20260927 13:58:47.744369 2563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:58:47.744278 2568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f49e6f8784401a1d39b0b262ceaec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35929 } }
I20260927 13:58:47.747428 2570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 749f49e6f8784401a1d39b0b262ceaec. Latest consensus state: current_term: 1 leader_uuid: "749f49e6f8784401a1d39b0b262ceaec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f49e6f8784401a1d39b0b262ceaec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35929 } } }
I20260927 13:58:47.747500 2569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "749f49e6f8784401a1d39b0b262ceaec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f49e6f8784401a1d39b0b262ceaec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35929 } } }
I20260927 13:58:47.747803 2570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [sys.catalog]: This master's current role is: LEADER
I20260927 13:58:47.747869 2569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [sys.catalog]: This master's current role is: LEADER
I20260927 13:58:47.749495 2576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:58:47.754154 2576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:58:47.754897 2563 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 13:58:47.773337 2576 catalog_manager.cc:1347] Generated new cluster ID: 38894c8a5c7a46f5a5e96756dd0b198c
I20260927 13:58:47.773603 2576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:58:47.790516 2576 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:58:47.792842 2576 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:58:47.802008 2576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec: Generated new TSK 0
I20260927 13:58:47.803268 2576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:58:48.523679 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.26.212:35947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35929
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:58:48 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790517528, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 13:58:48.972574 2608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:48.972983 2608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:48.973170 2608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:48.981328 2608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:48.981462 2608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:48.981667 2608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:58:48.992244 2608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.26.212:35947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:48.993677 2608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:48.995016 2608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:49.013037 2618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:49.014122 2619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:49.023139 2608 server_base.cc:1034] running on GCE node
W20260927 13:58:49.023584 2621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:49.024392 2608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:49.027412 2608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:49.028666 2608 hybrid_clock.cc:648] HybridClock initialized: now 1790517529028603 us; error 66 us; skew 500 ppm
I20260927 13:58:49.029006 2608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:49 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517529, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:49.032644 2608 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:58:49.040014 2608 webserver.cc:460] Webserver started at http://127.22.26.193:39923/ using document root <none> and password file <none>
I20260927 13:58:49.040779 2608 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:49.040918 2608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:58:49.041222 2608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:58:49 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517529, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:49.067121 2608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "733042dfedc54816b2cbac4bc100d1f6"
format_stamp: "Formatted at 2026-09-27 13:58:49 on dist-test-slave-gkks"
server_key: "4d9b41a31b16c398e33d829dd96bddcb"
server_key_iv: "e3bafce0ce555e7d6d12cfa96668df14"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:49.068333 2608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "733042dfedc54816b2cbac4bc100d1f6"
format_stamp: "Formatted at 2026-09-27 13:58:49 on dist-test-slave-gkks"
server_key: "4d9b41a31b16c398e33d829dd96bddcb"
server_key_iv: "e3bafce0ce555e7d6d12cfa96668df14"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:49.081867 2608 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.009s sys 0.002s
I20260927 13:58:49.111821 2628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:49.116499 2608 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.005s sys 0.006s
I20260927 13:58:49.116739 2608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "733042dfedc54816b2cbac4bc100d1f6"
format_stamp: "Formatted at 2026-09-27 13:58:49 on dist-test-slave-gkks"
server_key: "4d9b41a31b16c398e33d829dd96bddcb"
server_key_iv: "e3bafce0ce555e7d6d12cfa96668df14"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:49.117071 2608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:58:49.142761 2608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:49.147737 2608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:49.148108 2608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:49.151485 2608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:58:49.153384 2608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:58:49.153486 2608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:49.153687 2608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:58:49.153756 2608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:49.196704 2608 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:39555
I20260927 13:58:49.196730 2740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:39555 every 8 connection(s)
I20260927 13:58:49.198467 2608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:58:49.201182 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 2608
I20260927 13:58:49.201395 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:58:49.202296 22635 external_mini_cluster.cc:1428] Setting key 67b16b89313ce9b2c917a8b7f341f7e1
Sep 27 13:58:49 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517529, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
I20260927 13:58:49.242254 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:49.202658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:37263 (local address 127.22.26.254:35929)
0927 13:58:49.203778 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:49.203784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:49.218472 (+ 14688us) server_negotiation.cc:407] Connection header received
0927 13:58:49.221005 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:49.221013 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:49.221120 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:49.221246 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:49.224335 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:49.225509 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:49.228146 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:49.228878 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:49.228994 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:49.234054 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:49.234064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:49.234070 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:49.234129 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:49.238113 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:49.239273 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:49.239279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:49.239282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:49.239443 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:49.240055 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:49.240062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:49.240066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:49.240470 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:49.240584 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:49.241594 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0927 13:58:49.241824 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":889,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
I20260927 13:58:49.243582 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:49.201995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35929 (local address 127.22.26.193:37263)
0927 13:58:49.217816 (+ 15821us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:49.217891 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:49.220576 (+ 2685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:49.222238 (+ 1662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:49.222251 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:49.223057 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:49.224098 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:49.224128 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:49.225730 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:49.225737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:49.227921 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:49.227949 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:49.229375 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:49.229384 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:49.229692 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:49.230583 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:49.230619 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:49.233750 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:49.238328 (+ 4578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:49.238338 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:49.238357 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:49.239057 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:49.239594 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:49.239600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:49.239604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:49.239891 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:49.240699 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:49.240706 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:49.241125 (+ 419us) client_negotiation.cc:769] Sending connection context
0927 13:58:49.241434 (+ 309us) client_negotiation.cc:240] Negotiation successful
0927 13:58:49.241993 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15436,"thread_start_us":18925,"threads_started":1}
I20260927 13:58:49.246078 2741 heartbeater.cc:347] Connected to a master server at 127.22.26.254:35929
I20260927 13:58:49.246460 2741 heartbeater.cc:464] Registering TS with master...
I20260927 13:58:49.247498 2741 heartbeater.cc:511] Master 127.22.26.254:35929 requested a full tablet report, sending...
I20260927 13:58:49.250590 2528 ts_manager.cc:194] Registered new tserver with Master: 733042dfedc54816b2cbac4bc100d1f6 (127.22.26.193:39555)
I20260927 13:58:49.252874 2528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:37263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:58:49.283731 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.26.212:35947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35929
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:58:49 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517528, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
Sep 27 13:58:49 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517528, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
W20260927 13:58:49.755980 2747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:49.756521 2747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:49.756740 2747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:49.767951 2747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:49.768211 2747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:49.768503 2747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:58:49.784593 2747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.26.212:35947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:49.786443 2747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:49.788275 2747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:49.835665 2754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:49.841943 2755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:49.851413 2747 server_base.cc:1034] running on GCE node
W20260927 13:58:49.852593 2758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:49.853574 2747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:49.867429 2747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:49.883183 2747 hybrid_clock.cc:648] HybridClock initialized: now 1790517529883028 us; error 149 us; skew 500 ppm
I20260927 13:58:49.883697 2747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:49 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517529, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:49.888257 2747 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:58:49.912289 2747 webserver.cc:460] Webserver started at http://127.22.26.194:45467/ using document root <none> and password file <none>
I20260927 13:58:49.913600 2747 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:49.913851 2747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:58:49.914306 2747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:58:49 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517529, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:49.935253 2747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "79983e430b95440baafd650665bbf53b"
format_stamp: "Formatted at 2026-09-27 13:58:49 on dist-test-slave-gkks"
server_key: "5c1decf914da730fc9019daa67ab2d46"
server_key_iv: "a1a16d140ca03c913b4a66231d66914c"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:49.936419 2747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "79983e430b95440baafd650665bbf53b"
format_stamp: "Formatted at 2026-09-27 13:58:49 on dist-test-slave-gkks"
server_key: "5c1decf914da730fc9019daa67ab2d46"
server_key_iv: "a1a16d140ca03c913b4a66231d66914c"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:49.951653 2747 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.003s sys 0.005s
I20260927 13:58:49.995430 2764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:49.997694 2747 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.004s
I20260927 13:58:49.998041 2747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "79983e430b95440baafd650665bbf53b"
format_stamp: "Formatted at 2026-09-27 13:58:49 on dist-test-slave-gkks"
server_key: "5c1decf914da730fc9019daa67ab2d46"
server_key_iv: "a1a16d140ca03c913b4a66231d66914c"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:49.998425 2747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:58:50.049698 2747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:50.055186 2747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:50.055539 2747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:50.056774 2747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:58:50.058543 2747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:58:50.058637 2747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:50.058760 2747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:58:50.058828 2747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:50.103350 2747 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:37997
I20260927 13:58:50.103395 2883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:37997 every 8 connection(s)
I20260927 13:58:50.105283 2747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:58:50.113430 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 2747
I20260927 13:58:50.113659 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:58:50.114387 22635 external_mini_cluster.cc:1428] Setting key 7637c6d33ef05925e32bb7804d81076c
Sep 27 13:58:50 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517529, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
I20260927 13:58:50.151633 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:50.109509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:40439 (local address 127.22.26.254:35929)
0927 13:58:50.126586 (+ 17077us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:50.126593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:50.126619 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:58:50.128141 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:50.128148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:50.128251 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:50.128360 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:50.130676 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.131852 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.137626 (+ 5774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.138260 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.138389 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:50.144002 (+ 5613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:50.144013 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:50.144020 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:50.144072 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:50.148003 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:50.148776 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:50.148784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:50.148790 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:50.148980 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:50.149496 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:50.149501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:50.149505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:50.149883 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:50.150039 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:50.150875 (+ 836us) server_negotiation.cc:299] Negotiation successful
0927 13:58:50.151129 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16921,"thread_start_us":17859,"threads_started":1}
I20260927 13:58:50.157192 2886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:50.108871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35929 (local address 127.22.26.194:40439)
0927 13:58:50.124221 (+ 15350us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:50.124286 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:50.127651 (+ 3365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:50.128869 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:50.128883 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:50.129573 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:50.130428 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:50.130455 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.135249 (+ 4794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.135256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:50.137383 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:50.137412 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.138715 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.138722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:50.139019 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:50.139904 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:50.139935 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:50.143734 (+ 3799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:50.148185 (+ 4451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:50.148194 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:50.148213 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:50.148643 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:50.149087 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:50.149111 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:50.149114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:50.149363 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:50.150033 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:50.150041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:50.150507 (+ 466us) client_negotiation.cc:769] Sending connection context
0927 13:58:50.150771 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 13:58:50.155597 (+ 4826us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14142,"thread_start_us":13903,"threads_started":1}
I20260927 13:58:50.159420 2884 heartbeater.cc:347] Connected to a master server at 127.22.26.254:35929
I20260927 13:58:50.159765 2884 heartbeater.cc:464] Registering TS with master...
I20260927 13:58:50.160727 2884 heartbeater.cc:511] Master 127.22.26.254:35929 requested a full tablet report, sending...
I20260927 13:58:50.162489 2528 ts_manager.cc:194] Registered new tserver with Master: 79983e430b95440baafd650665bbf53b (127.22.26.194:37997)
I20260927 13:58:50.163816 2528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:40439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:58:50.183614 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.26.212:35947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35929
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:58:50.257165 2741 heartbeater.cc:503] Master 127.22.26.254:35929 was elected leader, sending a full tablet report...
W20260927 13:58:50.527875 2890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:50.528160 2890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:50.528256 2890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:50.535594 2890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:50.535705 2890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:50.535817 2890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:58:50.546162 2890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.26.212:35947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:50.547593 2890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:50.548916 2890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:50.561746 2895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:50.561764 2898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:50.562069 2896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:50.563035 2890 server_base.cc:1034] running on GCE node
I20260927 13:58:50.563637 2890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:50.564559 2890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:50.565757 2890 hybrid_clock.cc:648] HybridClock initialized: now 1790517530565708 us; error 45 us; skew 500 ppm
I20260927 13:58:50.566082 2890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:50 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517530, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:50.569309 2890 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:58:50.570711 2890 webserver.cc:460] Webserver started at http://127.22.26.195:42115/ using document root <none> and password file <none>
I20260927 13:58:50.571447 2890 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:50.571581 2890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:58:50.571893 2890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:58:50 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517530, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:50.588572 2890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "61d60dc2c93e478bb8245af98de27352"
format_stamp: "Formatted at 2026-09-27 13:58:50 on dist-test-slave-gkks"
server_key: "743b1f843f7eacd78fceedfb919046b4"
server_key_iv: "ae22991d06b28aeb104d4c0cecfffe3a"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:50.589363 2890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "61d60dc2c93e478bb8245af98de27352"
format_stamp: "Formatted at 2026-09-27 13:58:50 on dist-test-slave-gkks"
server_key: "743b1f843f7eacd78fceedfb919046b4"
server_key_iv: "ae22991d06b28aeb104d4c0cecfffe3a"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:50.594802 2890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:58:50.613337 2905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:50.615345 2890 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:58:50.615619 2890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "61d60dc2c93e478bb8245af98de27352"
format_stamp: "Formatted at 2026-09-27 13:58:50 on dist-test-slave-gkks"
server_key: "743b1f843f7eacd78fceedfb919046b4"
server_key_iv: "ae22991d06b28aeb104d4c0cecfffe3a"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:50.615934 2890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:58:50.633602 2890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:50.637462 2890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:50.637735 2890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:50.638742 2890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:58:50.640362 2890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:58:50.640437 2890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:50.640584 2890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:58:50.640638 2890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:50.674804 2890 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:35509
I20260927 13:58:50.674903 3017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:35509 every 8 connection(s)
I20260927 13:58:50.676152 2890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:58:50.678957 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 2890
I20260927 13:58:50.679194 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:58:50.679768 22635 external_mini_cluster.cc:1428] Setting key 5e1135ae155486fda5e4c7d1bbba6c9e
Sep 27 13:58:50 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517530, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:58:50.695900 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:50.678800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:60501 (local address 127.22.26.254:35929)
0927 13:58:50.679441 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:50.679449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:50.680542 (+ 1093us) server_negotiation.cc:407] Connection header received
0927 13:58:50.682379 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:50.682395 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:50.682478 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:50.682600 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:50.684133 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.684851 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.686363 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.686783 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.686881 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:50.690138 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:50.690143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:50.690146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:50.690177 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:50.692774 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:50.693580 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:50.693585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:50.693587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:50.693732 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:50.694260 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:50.694265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:50.694268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:50.694628 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:50.694769 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:50.695492 (+ 723us) server_negotiation.cc:299] Negotiation successful
0927 13:58:50.695658 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":301,"threads_started":1}
I20260927 13:58:50.696885 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:50.679448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35929 (local address 127.22.26.195:60501)
0927 13:58:50.680324 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:50.680385 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:50.682081 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:50.682866 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:50.682880 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:50.683387 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:50.683976 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:50.683994 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.684981 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.684984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:50.686214 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:50.686233 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.686960 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.686964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:50.687231 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:50.687866 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:50.687888 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:50.689953 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:50.692908 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:50.692914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:50.692925 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:50.693437 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:50.693842 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:50.693847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:50.693851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:50.694118 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:50.694764 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:50.694769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:50.695088 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 13:58:50.695388 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 13:58:50.695866 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":363,"threads_started":1}
I20260927 13:58:50.698165 3018 heartbeater.cc:347] Connected to a master server at 127.22.26.254:35929
I20260927 13:58:50.698414 3018 heartbeater.cc:464] Registering TS with master...
I20260927 13:58:50.699147 3018 heartbeater.cc:511] Master 127.22.26.254:35929 requested a full tablet report, sending...
I20260927 13:58:50.700606 2528 ts_manager.cc:194] Registered new tserver with Master: 61d60dc2c93e478bb8245af98de27352 (127.22.26.195:35509)
I20260927 13:58:50.701782 2528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:60501
I20260927 13:58:50.707337 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:58:50.709285 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 2608
Sep 27 13:58:50 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:58:50.726287 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:50.710309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.22.26.193:39555)
0927 13:58:50.710931 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:50.710937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:50.710962 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:58:50.711018 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:50.711023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:50.711259 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:50.711384 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:50.712759 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.713841 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.715283 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.715800 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.715893 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:50.720022 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:50.720036 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:50.720048 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:50.720093 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:50.723637 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:50.724382 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:50.724388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:50.724390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:50.724556 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:50.724878 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:50.724881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:50.724882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:50.725203 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:50.725429 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:50.725688 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 13:58:50.725880 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":327,"threads_started":1}
I20260927 13:58:50.727733 2736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:58:50.811447 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 2608
I20260927 13:58:50.827924 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 2747
Sep 27 13:58:50 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:58:50.844486 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:50.828841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.22.26.194:37997)
0927 13:58:50.829468 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:50.829475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:50.829501 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:58:50.829563 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:50.829570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:50.829802 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:50.829949 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:50.831188 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.832257 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.833571 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.834129 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.834229 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:50.838791 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:50.838804 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:50.838818 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:50.838860 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:50.841720 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:50.842405 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:50.842413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:50.842415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:50.842596 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:50.842927 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:50.842930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:50.842932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:50.843328 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:50.843693 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:50.843876 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 13:58:50.844094 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":330,"threads_started":1}
I20260927 13:58:50.845759 2879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:58:50.928799 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 2747
I20260927 13:58:50.950389 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 2890
Sep 27 13:58:50 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:58:50.968591 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:50.951593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.22.26.195:35509)
0927 13:58:50.952313 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:50.952320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:50.952468 (+ 148us) server_negotiation.cc:407] Connection header received
0927 13:58:50.952648 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:50.952655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:50.952892 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:50.953020 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:50.954498 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.955669 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.957158 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:50.957813 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:50.957926 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:50.962577 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:50.962590 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:50.962605 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:50.962646 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:50.965637 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:50.966304 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:50.966311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:50.966314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:50.966484 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:50.966887 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:50.966892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:50.966894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:50.967465 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:50.967748 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:50.968058 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 13:58:50.968272 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":373,"threads_started":1}
I20260927 13:58:50.970139 3013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:58:51.054147 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 2890
I20260927 13:58:51.069734 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 2496
I20260927 13:58:51.071059 2558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:58:51.178577 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 2496
I20260927 13:58:51.202733 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:35929
--webserver_interface=127.22.26.254
--webserver_port=34965
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:35929
--ranger_config_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.26.212:35947/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.26.212:35947/kms with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:58:51.537009 3030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:51.537350 3030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:51.537499 3030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:51.544831 3030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:58:51.544974 3030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:51.545078 3030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:51.545172 3030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:58:51.545248 3030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:58:51.555469 3030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.26.212:35947/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:35929
--ranger_config_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:35929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=34965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:51.556820 3030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:51.558182 3030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:51.570317 3035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:51.570233 3036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:51.570283 3038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:51.571200 3030 server_base.cc:1034] running on GCE node
I20260927 13:58:51.572028 3030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:51.573470 3030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:51.574704 3030 hybrid_clock.cc:648] HybridClock initialized: now 1790517531574654 us; error 44 us; skew 500 ppm
I20260927 13:58:51.575059 3030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:51 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517531, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:51.578287 3030 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:58:51.579838 3030 webserver.cc:460] Webserver started at http://127.22.26.254:34965/ using document root <none> and password file <none>
I20260927 13:58:51.580519 3030 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:51.580664 3030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 13:58:51 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517531, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:51.647316 3030 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:58:51.651065 3046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:51.652927 3030 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:58:51.653131 3030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "749f49e6f8784401a1d39b0b262ceaec"
format_stamp: "Formatted at 2026-09-27 13:58:47 on dist-test-slave-gkks"
server_key: "61687e281edabea2fe2ddaf89b08688d"
server_key_iv: "12cb32d47c96ba27d9b95edccf77c62f"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:51.654510 3030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:58:51.667649 3030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:51.671550 3030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:51.671790 3030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:51.690151 3030 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:35929
I20260927 13:58:51.690160 3097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:35929 every 8 connection(s)
I20260927 13:58:51.691684 3030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:58:51.696645 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 3030
I20260927 13:58:51.697499 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.26.212:35947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:39555
--local_ip_for_outbound_sockets=127.22.26.193
--tserver_master_addrs=127.22.26.254:35929
--webserver_port=39923
--webserver_interface=127.22.26.193
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.26.212:35947/kms with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:58:51.698575 3098 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:58:51.701758 3098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec: Bootstrap starting.
I20260927 13:58:51.720482 3098 log.cc:824] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec: Log is configured to *not* fsync() on all Append() calls
I20260927 13:58:51.729322 3098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:58:51.729888 3098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec: Bootstrap complete.
I20260927 13:58:51.738827 3098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f49e6f8784401a1d39b0b262ceaec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35929 } }
I20260927 13:58:51.739809 3098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749f49e6f8784401a1d39b0b262ceaec, State: Initialized, Role: FOLLOWER
I20260927 13:58:51.740404 3098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f49e6f8784401a1d39b0b262ceaec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35929 } }
I20260927 13:58:51.740640 3098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:58:51.740767 3098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:58:51.740937 3098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:58:51.744707 3098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f49e6f8784401a1d39b0b262ceaec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35929 } }
I20260927 13:58:51.745189 3098 leader_election.cc:302] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 749f49e6f8784401a1d39b0b262ceaec; no voters:
I20260927 13:58:51.745810 3098 leader_election.cc:288] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 13:58:51.745966 3102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:58:51.747186 3102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [term 2 LEADER]: Becoming Leader. State: Replica: 749f49e6f8784401a1d39b0b262ceaec, State: Running, Role: LEADER
I20260927 13:58:51.747846 3102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f49e6f8784401a1d39b0b262ceaec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35929 } }
I20260927 13:58:51.748211 3098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:58:51.750465 3104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 749f49e6f8784401a1d39b0b262ceaec. Latest consensus state: current_term: 2 leader_uuid: "749f49e6f8784401a1d39b0b262ceaec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f49e6f8784401a1d39b0b262ceaec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35929 } } }
I20260927 13:58:51.750905 3104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [sys.catalog]: This master's current role is: LEADER
I20260927 13:58:51.751362 3103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "749f49e6f8784401a1d39b0b262ceaec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f49e6f8784401a1d39b0b262ceaec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35929 } } }
I20260927 13:58:51.751688 3103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec [sys.catalog]: This master's current role is: LEADER
I20260927 13:58:51.754729 3109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:58:51.763063 3109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:58:51.765004 3109 catalog_manager.cc:1259] Loaded cluster ID: 38894c8a5c7a46f5a5e96756dd0b198c
I20260927 13:58:51.765206 3109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:58:51.770469 3109 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:58:51.772534 3109 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec: Loaded TSK: 0
I20260927 13:58:51.774636 3109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:58:51.789068 3098 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 13:58:52.156378 3100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:52.156876 3100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:52.157094 3100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:52.168920 3100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:52.169054 3100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:52.169246 3100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:58:52.185104 3100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.26.212:35947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:39555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=39923
--tserver_master_addrs=127.22.26.254:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:52.186820 3100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:52.188675 3100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:52.223338 3136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:52.223572 3137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:52.227761 3139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:52.228268 3100 server_base.cc:1034] running on GCE node
I20260927 13:58:52.229059 3100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:52.231374 3100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:52.238973 3100 hybrid_clock.cc:648] HybridClock initialized: now 1790517532238891 us; error 81 us; skew 500 ppm
I20260927 13:58:52.239512 3100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:52 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517532, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:52.249886 3100 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:58:52.254499 3100 webserver.cc:460] Webserver started at http://127.22.26.193:39923/ using document root <none> and password file <none>
I20260927 13:58:52.255571 3100 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:52.255790 3100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 13:58:52 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517532, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:52.301062 3100 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.006s sys 0.005s
I20260927 13:58:52.315258 3148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:52.317394 3100 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.008s sys 0.000s
I20260927 13:58:52.317600 3100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "733042dfedc54816b2cbac4bc100d1f6"
format_stamp: "Formatted at 2026-09-27 13:58:49 on dist-test-slave-gkks"
server_key: "4d9b41a31b16c398e33d829dd96bddcb"
server_key_iv: "e3bafce0ce555e7d6d12cfa96668df14"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:52.319164 3100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:58:52.375967 3100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:52.381554 3100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:52.381945 3100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:52.385190 3100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:58:52.387006 3100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:58:52.387189 3100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 13:58:52.387383 3100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:58:52.387539 3100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:52.471824 3100 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:39555
I20260927 13:58:52.471899 3260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:39555 every 8 connection(s)
I20260927 13:58:52.473814 3100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:58:52.480036 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 3100
I20260927 13:58:52.480970 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.26.212:35947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:37997
--local_ip_for_outbound_sockets=127.22.26.194
--tserver_master_addrs=127.22.26.254:35929
--webserver_port=45467
--webserver_interface=127.22.26.194
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.26.212:35947/kms with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:58:52 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517532, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:58:52.511412 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:52.483115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35929 (local address 127.22.26.193:47843)
0927 13:58:52.487373 (+ 4258us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:52.487435 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:52.489464 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:52.490972 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:52.490983 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:52.491577 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:52.492126 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:52.492143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:52.494428 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:52.494432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:52.495706 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:52.495718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:52.496724 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:52.496731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:52.497010 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:52.497840 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:52.497872 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:52.500615 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:52.505292 (+ 4677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:52.505306 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:52.505326 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:52.506061 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:52.506651 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:52.506657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:52.506660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:52.506888 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:52.507931 (+ 1043us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:52.507942 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:52.509071 (+ 1129us) client_negotiation.cc:769] Sending connection context
0927 13:58:52.509379 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 13:58:52.509977 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3871,"thread_start_us":3634,"threads_started":1}
I20260927 13:58:52.512485 3264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:52.488160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:47843 (local address 127.22.26.254:35929)
0927 13:58:52.488958 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:52.488978 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:52.489028 (+ 50us) server_negotiation.cc:407] Connection header received
0927 13:58:52.490081 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:52.490094 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:52.490449 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:52.490872 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:52.492323 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:52.494254 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:52.495855 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:52.496494 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:52.496689 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:52.500925 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:52.500936 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:52.500958 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:52.501013 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:52.505101 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:52.506246 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:52.506256 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:52.506265 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:52.506497 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:52.507029 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:52.507035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:52.507038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:52.507742 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:52.507965 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:52.509677 (+ 1712us) server_negotiation.cc:299] Negotiation successful
0927 13:58:52.510151 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":329,"threads_started":1}
I20260927 13:58:52.516320 3261 heartbeater.cc:347] Connected to a master server at 127.22.26.254:35929
I20260927 13:58:52.516661 3261 heartbeater.cc:464] Registering TS with master...
I20260927 13:58:52.517557 3261 heartbeater.cc:511] Master 127.22.26.254:35929 requested a full tablet report, sending...
W20260927 13:58:52.520862 3261 heartbeater.cc:644] Failed to heartbeat to 127.22.26.254:35929 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 13:58:52.521093 3261 heartbeater.cc:464] Registering TS with master...
I20260927 13:58:52.521364 3261 heartbeater.cc:511] Master 127.22.26.254:35929 requested a full tablet report, sending...
I20260927 13:58:52.522692 3261 heartbeater.cc:464] Registering TS with master...
I20260927 13:58:52.523025 3261 heartbeater.cc:511] Master 127.22.26.254:35929 requested a full tablet report, sending...
W20260927 13:58:52.524175 3261 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 13:58:52.867673 3265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:52.868042 3265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:52.868273 3265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:52.879544 3265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:52.879735 3265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:52.879948 3265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:58:52.891957 3265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.26.212:35947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:37997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=45467
--tserver_master_addrs=127.22.26.254:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:52.893745 3265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:52.895298 3265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:52.946548 3283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:52.948042 3280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:52.948060 3279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:52.950927 3265 server_base.cc:1034] running on GCE node
I20260927 13:58:52.951740 3265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:52.953620 3265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:52.954855 3265 hybrid_clock.cc:648] HybridClock initialized: now 1790517532954776 us; error 66 us; skew 500 ppm
I20260927 13:58:52.955356 3265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:52 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517532, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:52.965193 3265 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:58:52.967487 3265 webserver.cc:460] Webserver started at http://127.22.26.194:45467/ using document root <none> and password file <none>
I20260927 13:58:52.968461 3265 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:52.968636 3265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 13:58:52 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517532, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:52.995062 3265 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 13:58:52.998842 3292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:53.000805 3265 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:58:53.000998 3265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "79983e430b95440baafd650665bbf53b"
format_stamp: "Formatted at 2026-09-27 13:58:49 on dist-test-slave-gkks"
server_key: "5c1decf914da730fc9019daa67ab2d46"
server_key_iv: "a1a16d140ca03c913b4a66231d66914c"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:53.002391 3265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 27 13:58:53 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790517533, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:53.048203 3265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:53.053860 3265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:53.054157 3265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:53.055310 3265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:58:53.056921 3265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:58:53.057031 3265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:53.057128 3265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:58:53.057194 3265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:53.102041 3265 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:37997
I20260927 13:58:53.102131 3404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:37997 every 8 connection(s)
I20260927 13:58:53.104033 3265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:58:53.106120 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 3265
I20260927 13:58:53.106904 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.26.212:35947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:35509
--local_ip_for_outbound_sockets=127.22.26.195
--tserver_master_addrs=127.22.26.254:35929
--webserver_port=42115
--webserver_interface=127.22.26.195
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.26.212:35947/kms with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:58:53 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517532, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:58:53.147192 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:53.106783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:45913 (local address 127.22.26.254:35929)
0927 13:58:53.119886 (+ 13103us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:53.119892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:53.124005 (+ 4113us) server_negotiation.cc:407] Connection header received
0927 13:58:53.126926 (+ 2921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:53.126933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:53.127039 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:53.127146 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:53.129565 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:53.130710 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:53.133139 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:53.133813 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:53.133908 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:53.139356 (+ 5448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:53.139364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:53.139368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:53.139419 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:53.143163 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:53.144155 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:53.144160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:53.144164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:53.144314 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:53.144934 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:53.144939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:53.144953 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:53.145328 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:53.145483 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:53.146510 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0927 13:58:53.146752 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12970,"thread_start_us":12832,"threads_started":1}
I20260927 13:58:53.149891 3409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:53.107547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35929 (local address 127.22.26.194:45913)
0927 13:58:53.123770 (+ 16223us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:53.123834 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:53.126561 (+ 2727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:53.127748 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:53.127763 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:53.128438 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:53.129341 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:53.129368 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:53.130917 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:53.130922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:53.132970 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:53.132986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:53.134274 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:53.134282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:53.134575 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:53.135447 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:53.135480 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:53.139032 (+ 3552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:53.143341 (+ 4309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:53.143351 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:53.143371 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:53.144002 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:53.144476 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:53.144484 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:53.144487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:53.144761 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:53.145585 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:53.145593 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:53.146066 (+ 473us) client_negotiation.cc:769] Sending connection context
0927 13:58:53.147787 (+ 1721us) client_negotiation.cc:240] Negotiation successful
0927 13:58:53.148411 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15855,"thread_start_us":17114,"threads_started":1}
I20260927 13:58:53.152011 3406 heartbeater.cc:347] Connected to a master server at 127.22.26.254:35929
I20260927 13:58:53.152343 3406 heartbeater.cc:464] Registering TS with master...
I20260927 13:58:53.153308 3406 heartbeater.cc:511] Master 127.22.26.254:35929 requested a full tablet report, sending...
I20260927 13:58:53.155501 3063 ts_manager.cc:194] Registered new tserver with Master: 79983e430b95440baafd650665bbf53b (127.22.26.194:37997)
I20260927 13:58:53.157627 3063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:45913
Sep 27 13:58:53 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517533, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:53.524364 3261 heartbeater.cc:464] Registering TS with master...
I20260927 13:58:53.524920 3261 heartbeater.cc:511] Master 127.22.26.254:35929 requested a full tablet report, sending...
I20260927 13:58:53.526448 3063 ts_manager.cc:194] Registered new tserver with Master: 733042dfedc54816b2cbac4bc100d1f6 (127.22.26.193:39555)
I20260927 13:58:53.527676 3063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:47843
W20260927 13:58:53.551362 3410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:53.551869 3410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:53.552078 3410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:53.563666 3410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:53.563889 3410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:53.564122 3410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:58:53.580052 3410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.26.212:35947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:35509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=42115
--tserver_master_addrs=127.22.26.254:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:53.582067 3410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:53.583925 3410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:53.619877 3416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:53.623482 3415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:53.624696 3410 server_base.cc:1034] running on GCE node
W20260927 13:58:53.627365 3419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:53.628237 3410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:53.639429 3410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:53.640789 3410 hybrid_clock.cc:648] HybridClock initialized: now 1790517533640715 us; error 76 us; skew 500 ppm
I20260927 13:58:53.641268 3410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:53 dist-test-slave-gkks krb5kdc[2052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517533, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:53.645521 3410 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:58:53.669606 3410 webserver.cc:460] Webserver started at http://127.22.26.195:42115/ using document root <none> and password file <none>
I20260927 13:58:53.670674 3410 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:53.670899 3410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 13:58:53 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517533, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:53.708009 3410 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 13:58:53.717947 3426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:53.719959 3410 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.007s
I20260927 13:58:53.720189 3410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "61d60dc2c93e478bb8245af98de27352"
format_stamp: "Formatted at 2026-09-27 13:58:50 on dist-test-slave-gkks"
server_key: "743b1f843f7eacd78fceedfb919046b4"
server_key_iv: "ae22991d06b28aeb104d4c0cecfffe3a"
server_key_version: "kuduclusterkey@0"
I20260927 13:58:53.721686 3410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:58:53.763928 3410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:53.769465 3410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:53.769871 3410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:53.777415 3410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:58:53.788650 3410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:58:53.788862 3410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 13:58:53.789127 3410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:58:53.789256 3410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:58:53 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517533, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for HTTP/127.22.26.212@KRBTEST.COM
I20260927 13:58:54.162222 3406 heartbeater.cc:503] Master 127.22.26.254:35929 was elected leader, sending a full tablet report...
I20260927 13:58:54.331576 3410 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:35509
I20260927 13:58:54.331593 3546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:35509 every 8 connection(s)
I20260927 13:58:54.333092 3410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:58:54.339656 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 3410
Sep 27 13:58:54 dist-test-slave-gkks krb5kdc[2052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517533, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:58:54.378799 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.341496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36350 (local address 127.22.26.254:35929)
0927 13:58:54.356609 (+ 15113us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.356618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.357390 (+ 772us) server_negotiation.cc:407] Connection header received
0927 13:58:54.357474 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.357481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.357582 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.357687 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:54.360315 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.361531 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.363620 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.364281 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.364433 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.368418 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:54.368426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:54.368430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:54.368471 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:54.372570 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:54.373470 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:54.373478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:54.373485 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:54.373649 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:54.376424 (+ 2775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:54.376431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:54.376434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:54.376831 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:54.377092 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:54.378144 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0927 13:58:54.378372 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":14951,"thread_start_us":11559,"threads_started":1}
I20260927 13:58:54.381515 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.341475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:50813 (local address 127.22.26.254:35929)
0927 13:58:54.353022 (+ 11547us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.353032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.357079 (+ 4047us) server_negotiation.cc:407] Connection header received
0927 13:58:54.360250 (+ 3171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.360255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.360340 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.360453 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:54.362853 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.364142 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.366680 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.367463 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.367569 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.373608 (+ 6039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:54.373615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:54.373619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:54.373664 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:54.376251 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:54.377336 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:54.377342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:54.377345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:54.377484 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:54.378622 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:54.378626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:54.378629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:54.379045 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:54.379188 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:54.380898 (+ 1710us) server_negotiation.cc:299] Negotiation successful
0927 13:58:54.381117 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1245,"server-negotiator.queue_time_us":11380,"thread_start_us":14873,"threads_started":1}
I20260927 13:58:54.383533 3552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.343445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35929 (local address 127.22.26.195:50813)
0927 13:58:54.355763 (+ 12318us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:54.355816 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:54.359656 (+ 3840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:54.360823 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:54.360836 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:54.361476 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:54.362622 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.362648 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.364354 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.364359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.366501 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.366518 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.367960 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.367968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.368300 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.369067 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:54.369105 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:54.372575 (+ 3470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:54.376577 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:54.376588 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:54.376607 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:54.377214 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:54.377624 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:54.377628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:54.377632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:54.377925 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:54.379305 (+ 1380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:54.379315 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:54.379833 (+ 518us) client_negotiation.cc:769] Sending connection context
0927 13:58:54.381778 (+ 1945us) client_negotiation.cc:240] Negotiation successful
0927 13:58:54.382394 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11989,"thread_start_us":11726,"threads_started":1}
I20260927 13:58:54.384979 3547 heartbeater.cc:347] Connected to a master server at 127.22.26.254:35929
I20260927 13:58:54.385298 3547 heartbeater.cc:464] Registering TS with master...
I20260927 13:58:54.386045 3547 heartbeater.cc:511] Master 127.22.26.254:35929 requested a full tablet report, sending...
I20260927 13:58:54.387521 3063 ts_manager.cc:194] Registered new tserver with Master: 61d60dc2c93e478bb8245af98de27352 (127.22.26.195:35509)
I20260927 13:58:54.388360 3063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:50813
I20260927 13:58:54.394655 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:58:54.413183 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.398096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36366 (local address 127.22.26.254:35929)
0927 13:58:54.398358 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.398365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.398758 (+ 393us) server_negotiation.cc:407] Connection header received
0927 13:58:54.399006 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.399011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.399144 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.399244 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:54.401240 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.401950 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.403874 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.404275 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.404373 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.407712 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:54.407717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:54.407720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:54.407754 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:54.410487 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:54.411285 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:54.411289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:54.411292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:54.411428 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:54.411843 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:54.411847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:54.411849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:54.412136 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:54.412328 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:54.412714 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 13:58:54.412881 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:58:54.418267 3063 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:58:54.426062 3063 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:58:54.478012 3562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.441787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:39555 (local address 127.0.0.1:55674)
0927 13:58:54.457800 (+ 16013us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:54.457854 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:54.458111 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:54.468196 (+ 10085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:54.468202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:54.468229 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:58:54.468567 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.468581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.470703 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.470712 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.474642 (+ 3930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.474656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.476584 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.476589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.476925 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.477051 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 13:58:54.477201 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 13:58:54.477510 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15648,"mutex_wait_us":1618,"thread_start_us":15476,"threads_started":1}
I20260927 13:58:54.478012 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.442999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.22.26.195:35509)
0927 13:58:54.465010 (+ 22011us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.465023 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.465064 (+ 41us) server_negotiation.cc:407] Connection header received
0927 13:58:54.465186 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.465196 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.465506 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.465773 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:58:54.468597 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.470270 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.474362 (+ 4092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.475959 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.476102 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.476355 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:54.477417 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0927 13:58:54.477604 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21813,"thread_start_us":21641,"threads_started":1}
I20260927 13:58:54.478808 3566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.442216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:35509 (local address 127.0.0.1:42984)
0927 13:58:54.463671 (+ 21455us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:54.463696 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:54.463860 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:54.466169 (+ 2309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:54.466177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:54.466217 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:58:54.466783 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.466800 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.470483 (+ 3683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.470489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.474143 (+ 3654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.474160 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.476547 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.476555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.476902 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.477047 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 13:58:54.477205 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 13:58:54.478342 (+ 1137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21299,"mutex_wait_us":1138,"thread_start_us":29031,"threads_started":1}
I20260927 13:58:54.479940 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.442195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.22.26.193:39555)
0927 13:58:54.467500 (+ 25305us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.467511 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.467547 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:58:54.467659 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.467670 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.467963 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.468218 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:58:54.468787 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.470532 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.474829 (+ 4297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.476335 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.478772 (+ 2437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.479026 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:54.479420 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 13:58:54.479591 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25100,"thread_start_us":30593,"threads_started":1}
I20260927 13:58:54.482999 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.443702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:37997 (local address 127.0.0.1:40402)
0927 13:58:54.471533 (+ 27831us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:54.471555 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:54.471689 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:54.473130 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:54.473134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:54.473160 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:58:54.473630 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.473643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.476936 (+ 3293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.476940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.479237 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.479254 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.482101 (+ 2847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.482107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.482290 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.482332 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 13:58:54.482444 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 13:58:54.482583 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27690,"mutex_wait_us":21,"thread_start_us":19640,"threads_started":1}
I20260927 13:58:54.483017 3567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.444404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40402 (local address 127.22.26.194:37997)
0927 13:58:54.464463 (+ 20059us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.464473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.471690 (+ 7217us) server_negotiation.cc:407] Connection header received
0927 13:58:54.471820 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.471828 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.472931 (+ 1103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.473136 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:58:54.473852 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.476773 (+ 2921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.479427 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.481849 (+ 2422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.481994 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.482212 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:54.482510 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 13:58:54.482671 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19855,"thread_start_us":19681,"threads_started":1}
I20260927 13:58:54.483523 3475 tablet_service.cc:1467] Processing CreateTablet for tablet 7f88f450a0144ddc93f6ab60bf656eac (DEFAULT_TABLE table=test-table [id=e2beb137ee5541c489eb47339c5bfe18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:58:54.485105 3475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f88f450a0144ddc93f6ab60bf656eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:58:54.487057 3340 tablet_service.cc:1467] Processing CreateTablet for tablet 7f88f450a0144ddc93f6ab60bf656eac (DEFAULT_TABLE table=test-table [id=e2beb137ee5541c489eb47339c5bfe18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:58:54.488788 3340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f88f450a0144ddc93f6ab60bf656eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:58:54.491477 3196 tablet_service.cc:1467] Processing CreateTablet for tablet 7f88f450a0144ddc93f6ab60bf656eac (DEFAULT_TABLE table=test-table [id=e2beb137ee5541c489eb47339c5bfe18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:58:54.492997 3196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f88f450a0144ddc93f6ab60bf656eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:58:54.508642 3568 tablet_bootstrap.cc:492] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352: Bootstrap starting.
I20260927 13:58:54.514416 3569 tablet_bootstrap.cc:492] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b: Bootstrap starting.
I20260927 13:58:54.515867 3568 tablet_bootstrap.cc:654] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352: Neither blocks nor log segments found. Creating new log.
I20260927 13:58:54.517319 3568 log.cc:824] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352: Log is configured to *not* fsync() on all Append() calls
I20260927 13:58:54.518597 3570 tablet_bootstrap.cc:492] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6: Bootstrap starting.
I20260927 13:58:54.520116 3569 tablet_bootstrap.cc:654] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b: Neither blocks nor log segments found. Creating new log.
I20260927 13:58:54.521438 3569 log.cc:824] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:58:54.522161 3570 tablet_bootstrap.cc:654] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6: Neither blocks nor log segments found. Creating new log.
I20260927 13:58:54.523437 3570 log.cc:824] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6: Log is configured to *not* fsync() on all Append() calls
I20260927 13:58:54.525022 3569 tablet_bootstrap.cc:492] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b: No bootstrap required, opened a new log
I20260927 13:58:54.525173 3568 tablet_bootstrap.cc:492] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352: No bootstrap required, opened a new log
I20260927 13:58:54.525341 3569 ts_tablet_manager.cc:1397] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 13:58:54.525593 3568 ts_tablet_manager.cc:1397] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20260927 13:58:54.531025 3261 heartbeater.cc:503] Master 127.22.26.254:35929 was elected leader, sending a full tablet report...
I20260927 13:58:54.533119 3570 tablet_bootstrap.cc:492] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6: No bootstrap required, opened a new log
I20260927 13:58:54.533384 3570 ts_tablet_manager.cc:1397] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260927 13:58:54.535471 3569 raft_consensus.cc:348] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733042dfedc54816b2cbac4bc100d1f6" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39555 } } peers { permanent_uuid: "79983e430b95440baafd650665bbf53b" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37997 } } peers { permanent_uuid: "61d60dc2c93e478bb8245af98de27352" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 35509 } }
I20260927 13:58:54.535940 3569 raft_consensus.cc:374] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:58:54.536070 3569 raft_consensus.cc:729] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79983e430b95440baafd650665bbf53b, State: Initialized, Role: FOLLOWER
I20260927 13:58:54.536736 3569 consensus_queue.cc:260] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733042dfedc54816b2cbac4bc100d1f6" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39555 } } peers { permanent_uuid: "79983e430b95440baafd650665bbf53b" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37997 } } peers { permanent_uuid: "61d60dc2c93e478bb8245af98de27352" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 35509 } }
I20260927 13:58:54.543291 3568 raft_consensus.cc:348] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733042dfedc54816b2cbac4bc100d1f6" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39555 } } peers { permanent_uuid: "79983e430b95440baafd650665bbf53b" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37997 } } peers { permanent_uuid: "61d60dc2c93e478bb8245af98de27352" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 35509 } }
I20260927 13:58:54.543905 3568 raft_consensus.cc:374] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:58:54.544102 3568 raft_consensus.cc:729] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d60dc2c93e478bb8245af98de27352, State: Initialized, Role: FOLLOWER
I20260927 13:58:54.543923 3570 raft_consensus.cc:348] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733042dfedc54816b2cbac4bc100d1f6" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39555 } } peers { permanent_uuid: "79983e430b95440baafd650665bbf53b" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37997 } } peers { permanent_uuid: "61d60dc2c93e478bb8245af98de27352" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 35509 } }
I20260927 13:58:54.544888 3570 raft_consensus.cc:374] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:58:54.545025 3570 raft_consensus.cc:729] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733042dfedc54816b2cbac4bc100d1f6, State: Initialized, Role: FOLLOWER
I20260927 13:58:54.544859 3568 consensus_queue.cc:260] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733042dfedc54816b2cbac4bc100d1f6" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39555 } } peers { permanent_uuid: "79983e430b95440baafd650665bbf53b" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37997 } } peers { permanent_uuid: "61d60dc2c93e478bb8245af98de27352" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 35509 } }
I20260927 13:58:54.545635 3569 ts_tablet_manager.cc:1428] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b: Time spent starting tablet: real 0.020s user 0.005s sys 0.014s
I20260927 13:58:54.546293 3570 consensus_queue.cc:260] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733042dfedc54816b2cbac4bc100d1f6" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39555 } } peers { permanent_uuid: "79983e430b95440baafd650665bbf53b" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37997 } } peers { permanent_uuid: "61d60dc2c93e478bb8245af98de27352" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 35509 } }
I20260927 13:58:54.552673 3547 heartbeater.cc:503] Master 127.22.26.254:35929 was elected leader, sending a full tablet report...
I20260927 13:58:54.553697 3570 ts_tablet_manager.cc:1428] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6: Time spent starting tablet: real 0.020s user 0.014s sys 0.006s
I20260927 13:58:54.555058 3568 ts_tablet_manager.cc:1428] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352: Time spent starting tablet: real 0.029s user 0.016s sys 0.005s
I20260927 13:58:54.555671 3576 raft_consensus.cc:482] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:58:54.555979 3576 raft_consensus.cc:504] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733042dfedc54816b2cbac4bc100d1f6" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39555 } } peers { permanent_uuid: "79983e430b95440baafd650665bbf53b" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37997 } } peers { permanent_uuid: "61d60dc2c93e478bb8245af98de27352" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 35509 } }
I20260927 13:58:54.557808 3576 leader_election.cc:288] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79983e430b95440baafd650665bbf53b (127.22.26.194:37997), 61d60dc2c93e478bb8245af98de27352 (127.22.26.195:35509)
I20260927 13:58:54.562712 3575 raft_consensus.cc:482] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:58:54.563032 3575 raft_consensus.cc:504] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733042dfedc54816b2cbac4bc100d1f6" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39555 } } peers { permanent_uuid: "79983e430b95440baafd650665bbf53b" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37997 } } peers { permanent_uuid: "61d60dc2c93e478bb8245af98de27352" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 35509 } }
I20260927 13:58:54.566581 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.558189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:44987 (local address 127.22.26.195:35509)
0927 13:58:54.558353 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.558359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.559148 (+ 789us) server_negotiation.cc:407] Connection header received
0927 13:58:54.559422 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.559431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.559538 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.559632 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:58:54.560440 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.561524 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.564076 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.565300 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.565431 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.565528 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:54.566159 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 13:58:54.566299 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:58:54.566910 3567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.558214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:54841 (local address 127.22.26.194:37997)
0927 13:58:54.558482 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.558491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.559906 (+ 1415us) server_negotiation.cc:407] Connection header received
0927 13:58:54.560043 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.560050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.560127 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.560228 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:58:54.560805 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.561848 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.564478 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.565678 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.565802 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.565867 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:54.566426 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 13:58:54.566574 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 13:58:54.567441 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.558059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:35509 (local address 127.22.26.193:44987)
0927 13:58:54.559021 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:54.559042 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:54.559253 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:54.559818 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:54.559822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:54.559842 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:58:54.560299 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.560315 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.561687 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.561693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.563887 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.563902 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.565716 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.565724 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.565946 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.565981 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:58:54.566808 (+ 827us) client_negotiation.cc:240] Negotiation successful
0927 13:58:54.566967 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":25,"thread_start_us":638,"threads_started":1}
I20260927 13:58:54.567953 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.558912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:37997 (local address 127.22.26.193:54841)
0927 13:58:54.559799 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:54.559817 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:54.559932 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:54.560262 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:54.560266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:54.560293 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:58:54.560639 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.560649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.561984 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.561994 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.564298 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.564313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.566076 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.566082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.566250 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.566280 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:58:54.567303 (+ 1023us) client_negotiation.cc:240] Negotiation successful
0927 13:58:54.567447 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":52,"thread_start_us":585,"threads_started":1}
I20260927 13:58:54.568373 3495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f88f450a0144ddc93f6ab60bf656eac" candidate_uuid: "733042dfedc54816b2cbac4bc100d1f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d60dc2c93e478bb8245af98de27352" is_pre_election: true
I20260927 13:58:54.568877 3495 raft_consensus.cc:2454] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733042dfedc54816b2cbac4bc100d1f6 in term 0.
I20260927 13:58:54.569049 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f88f450a0144ddc93f6ab60bf656eac" candidate_uuid: "733042dfedc54816b2cbac4bc100d1f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79983e430b95440baafd650665bbf53b" is_pre_election: true
I20260927 13:58:54.569563 3360 raft_consensus.cc:2454] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733042dfedc54816b2cbac4bc100d1f6 in term 0.
I20260927 13:58:54.569669 3575 leader_election.cc:288] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 733042dfedc54816b2cbac4bc100d1f6 (127.22.26.193:39555), 79983e430b95440baafd650665bbf53b (127.22.26.194:37997)
I20260927 13:58:54.570302 3150 leader_election.cc:302] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 733042dfedc54816b2cbac4bc100d1f6, 79983e430b95440baafd650665bbf53b; no voters:
I20260927 13:58:54.570751 3576 raft_consensus.cc:2781] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:58:54.570894 3576 raft_consensus.cc:482] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:58:54.570976 3576 raft_consensus.cc:3037] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:58:54.574226 3552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.565177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:39555 (local address 127.22.26.195:43253)
0927 13:58:54.565582 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:54.565600 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:54.565698 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:54.566564 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:54.566575 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:54.566604 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:58:54.566798 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.566808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.568363 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.568371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.571567 (+ 3196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.571585 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.573480 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.573486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.573704 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.573743 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 13:58:54.573830 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 13:58:54.573969 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292}
I20260927 13:58:54.574301 3576 raft_consensus.cc:504] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733042dfedc54816b2cbac4bc100d1f6" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39555 } } peers { permanent_uuid: "79983e430b95440baafd650665bbf53b" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37997 } } peers { permanent_uuid: "61d60dc2c93e478bb8245af98de27352" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 35509 } }
I20260927 13:58:54.574878 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.566035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:43253 (local address 127.22.26.193:39555)
0927 13:58:54.566213 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.566219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.566234 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:58:54.566284 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.566287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.566351 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.566429 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:58:54.567017 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.568178 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.571792 (+ 3614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.573133 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.573301 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.573386 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:54.574481 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0927 13:58:54.574621 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:58:54.575723 3216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f88f450a0144ddc93f6ab60bf656eac" candidate_uuid: "61d60dc2c93e478bb8245af98de27352" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733042dfedc54816b2cbac4bc100d1f6" is_pre_election: true
I20260927 13:58:54.576220 3216 raft_consensus.cc:2379] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61d60dc2c93e478bb8245af98de27352 in current term 1: Already voted for candidate 733042dfedc54816b2cbac4bc100d1f6 in this term.
I20260927 13:58:54.576354 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f88f450a0144ddc93f6ab60bf656eac" candidate_uuid: "733042dfedc54816b2cbac4bc100d1f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79983e430b95440baafd650665bbf53b"
I20260927 13:58:54.576560 3360 raft_consensus.cc:3037] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:58:54.579950 3567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.571384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:38877 (local address 127.22.26.194:37997)
0927 13:58:54.571542 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.571548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.572068 (+ 520us) server_negotiation.cc:407] Connection header received
0927 13:58:54.572311 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.572317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.572383 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.572440 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:58:54.573286 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.574414 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.578054 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.578945 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.579160 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.579231 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:54.579564 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 13:58:54.579690 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:58:54.580067 3579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.570159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:37997 (local address 127.22.26.195:38877)
0927 13:58:54.571978 (+ 1819us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:54.571997 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:54.572164 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:54.572620 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:54.572626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:54.572651 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:58:54.573024 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.573036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.574600 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.574605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.577905 (+ 3300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:54.577917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.579175 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.579180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:54.579356 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.579379 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 13:58:54.579541 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 13:58:54.579665 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1684,"thread_start_us":5203,"threads_started":1}
I20260927 13:58:54.580124 3495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f88f450a0144ddc93f6ab60bf656eac" candidate_uuid: "733042dfedc54816b2cbac4bc100d1f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d60dc2c93e478bb8245af98de27352"
I20260927 13:58:54.580350 3495 raft_consensus.cc:3037] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:58:54.583483 3495 raft_consensus.cc:2454] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733042dfedc54816b2cbac4bc100d1f6 in term 1.
I20260927 13:58:54.583634 3359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f88f450a0144ddc93f6ab60bf656eac" candidate_uuid: "61d60dc2c93e478bb8245af98de27352" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79983e430b95440baafd650665bbf53b" is_pre_election: true
I20260927 13:58:54.583983 3149 leader_election.cc:302] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d60dc2c93e478bb8245af98de27352, 733042dfedc54816b2cbac4bc100d1f6; no voters:
I20260927 13:58:54.584121 3360 raft_consensus.cc:2454] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733042dfedc54816b2cbac4bc100d1f6 in term 1.
I20260927 13:58:54.584571 3576 leader_election.cc:288] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [CANDIDATE]: Term 1 election: Requested vote from peers 79983e430b95440baafd650665bbf53b (127.22.26.194:37997), 61d60dc2c93e478bb8245af98de27352 (127.22.26.195:35509)
I20260927 13:58:54.584800 3576 raft_consensus.cc:2781] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:58:54.584673 3428 leader_election.cc:302] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61d60dc2c93e478bb8245af98de27352; no voters: 733042dfedc54816b2cbac4bc100d1f6, 79983e430b95440baafd650665bbf53b
I20260927 13:58:54.584934 3576 raft_consensus.cc:686] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 1 LEADER]: Becoming Leader. State: Replica: 733042dfedc54816b2cbac4bc100d1f6, State: Running, Role: LEADER
I20260927 13:58:54.585458 3575 raft_consensus.cc:2726] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 13:58:54.585477 3576 consensus_queue.cc:237] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733042dfedc54816b2cbac4bc100d1f6" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39555 } } peers { permanent_uuid: "79983e430b95440baafd650665bbf53b" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37997 } } peers { permanent_uuid: "61d60dc2c93e478bb8245af98de27352" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 35509 } }
I20260927 13:58:54.590500 3063 catalog_manager.cc:5495] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 733042dfedc54816b2cbac4bc100d1f6 (127.22.26.193). New cstate: current_term: 1 leader_uuid: "733042dfedc54816b2cbac4bc100d1f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733042dfedc54816b2cbac4bc100d1f6" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79983e430b95440baafd650665bbf53b" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61d60dc2c93e478bb8245af98de27352" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 35509 } health_report { overall_health: UNKNOWN } } }
W20260927 13:58:54.601838 3548 tablet.cc:2369] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:58:54.606474 3407 tablet.cc:2369] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:58:54.618285 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.611672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55676 (local address 127.22.26.193:39555)
0927 13:58:54.611853 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.611860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.611936 (+ 76us) server_negotiation.cc:407] Connection header received
0927 13:58:54.612153 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.612159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.612272 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.612431 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:58:54.612860 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.613958 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.615531 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.616176 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.616302 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.616873 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:58:54.617611 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:58:54.617755 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:54.617904 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 13:58:54.617987 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:58:54.628311 3360 raft_consensus.cc:1261] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b [term 1 FOLLOWER]: Refusing update from remote peer 733042dfedc54816b2cbac4bc100d1f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:58:54.628396 3495 raft_consensus.cc:1261] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [term 1 FOLLOWER]: Refusing update from remote peer 733042dfedc54816b2cbac4bc100d1f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:58:54.629259 3580 consensus_queue.cc:1035] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79983e430b95440baafd650665bbf53b" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:58:54.629673 3576 consensus_queue.cc:1035] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d60dc2c93e478bb8245af98de27352" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:58:54.647207 3584 mvcc.cc:205] Tried to move back new op lower bound from 7333959821826936832 to 7333959821666996224. Current Snapshot: MvccSnapshot[applied={T|T < 7333959821826936832}]
I20260927 13:58:54.649055 3585 mvcc.cc:205] Tried to move back new op lower bound from 7333959821826936832 to 7333959821666996224. Current Snapshot: MvccSnapshot[applied={T|T < 7333959821826936832}]
I20260927 13:58:54.663642 3062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:58:54.663966 3062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:58:54.667745 3062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 749f49e6f8784401a1d39b0b262ceaec: Sending DeleteTablet for 3 replicas of tablet 7f88f450a0144ddc93f6ab60bf656eac
I20260927 13:58:54.668893 3196 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f88f450a0144ddc93f6ab60bf656eac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:58:54 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:55674
I20260927 13:58:54.668883 3340 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f88f450a0144ddc93f6ab60bf656eac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:58:54 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:40402
I20260927 13:58:54.671613 3475 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f88f450a0144ddc93f6ab60bf656eac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:58:54 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:42984
I20260927 13:58:54.679029 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 3100
I20260927 13:58:54.696230 3594 tablet_replica.cc:330] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352: stopping tablet replica
I20260927 13:58:54.706005 3593 tablet_replica.cc:330] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b: stopping tablet replica
I20260927 13:58:54.706740 3593 raft_consensus.cc:2229] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:58:54.707798 3593 raft_consensus.cc:2258] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:58:54.708925 3592 tablet_replica.cc:330] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6: stopping tablet replica
I20260927 13:58:54.712405 3594 raft_consensus.cc:2229] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:58:54.713140 3594 raft_consensus.cc:2258] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:58:54.715674 3594 ts_tablet_manager.cc:1905] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:58:54.716574 3593 ts_tablet_manager.cc:1905] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:58:54.712055 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.680991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.22.26.193:39555)
0927 13:58:54.681275 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.681285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.681309 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:58:54.681385 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.681394 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.681508 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.681629 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:54.684415 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.686120 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.695450 (+ 9330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.696603 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.696765 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.701573 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:54.701598 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:54.701627 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:54.701692 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:54.707212 (+ 5520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:54.708301 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:54.708311 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:54.708316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:54.708499 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:54.709026 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:54.709031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:54.709035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:54.709618 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:54.709987 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:54.711441 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0927 13:58:54.711692 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:58:54.719604 3592 raft_consensus.cc:2229] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:58:54.720005 3256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:58:54.723536 3593 ts_tablet_manager.cc:1918] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:58:54.730011 3593 log.cc:1196] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/7f88f450a0144ddc93f6ab60bf656eac
I20260927 13:58:54.730304 3594 ts_tablet_manager.cc:1918] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:58:54.730531 3594 log.cc:1196] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/7f88f450a0144ddc93f6ab60bf656eac
I20260927 13:58:54.730716 3593 ts_tablet_manager.cc:1939] T 7f88f450a0144ddc93f6ab60bf656eac P 79983e430b95440baafd650665bbf53b: Deleting consensus metadata
I20260927 13:58:54.731339 3594 ts_tablet_manager.cc:1939] T 7f88f450a0144ddc93f6ab60bf656eac P 61d60dc2c93e478bb8245af98de27352: Deleting consensus metadata
I20260927 13:58:54.732913 3048 catalog_manager.cc:4841] TS 79983e430b95440baafd650665bbf53b (127.22.26.194:37997): tablet 7f88f450a0144ddc93f6ab60bf656eac (table test-table [id=e2beb137ee5541c489eb47339c5bfe18]) successfully deleted
I20260927 13:58:54.733143 3047 catalog_manager.cc:4841] TS 61d60dc2c93e478bb8245af98de27352 (127.22.26.195:35509): tablet 7f88f450a0144ddc93f6ab60bf656eac (table test-table [id=e2beb137ee5541c489eb47339c5bfe18]) successfully deleted
I20260927 13:58:54.831293 3592 raft_consensus.cc:2258] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:58:54.833797 3592 ts_tablet_manager.cc:1905] T 7f88f450a0144ddc93f6ab60bf656eac P 733042dfedc54816b2cbac4bc100d1f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:58:54.839792 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 3100
I20260927 13:58:54.858335 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 3265
W20260927 13:58:54.858481 3049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.26.193:39555 (error 108)
I20260927 13:58:54.862331 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.861823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:39555 (local address 127.0.0.1:55692)
0927 13:58:54.862023 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:54.862149 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:39555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 13:58:54.862646 3049 catalog_manager.cc:4584] TS 733042dfedc54816b2cbac4bc100d1f6 (127.22.26.193:39555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f88f450a0144ddc93f6ab60bf656eac: Network error: Client connection negotiation failed: client connection to 127.22.26.193:39555: connect: Connection refused (error 111)
I20260927 13:58:54.873023 3567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.859297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.22.26.194:37997)
0927 13:58:54.859493 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.859499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.860014 (+ 515us) server_negotiation.cc:407] Connection header received
0927 13:58:54.860088 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.860091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.860167 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.860287 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:54.861619 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.862497 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.863953 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.864366 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.864496 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.867068 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:54.867119 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:54.867138 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:54.867182 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:54.870126 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:54.870880 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:54.870888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:54.870890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:54.871024 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:54.871433 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:54.871438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:54.871441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:54.871802 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:54.872119 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:54.872428 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 13:58:54.872675 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:58:54.873796 3400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:58:54.916349 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.915788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:39555 (local address 127.0.0.1:55706)
0927 13:58:54.916032 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:54.916167 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:39555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 13:58:54.917546 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.917121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:39555 (local address 127.0.0.1:55718)
0927 13:58:54.917283 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:54.917385 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:39555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 13:58:54.968364 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 3265
I20260927 13:58:54.978156 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.977684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:39555 (local address 127.0.0.1:55724)
0927 13:58:54.977889 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:54.977974 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:39555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 13:58:54.979544 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.979189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:39555 (local address 127.0.0.1:55726)
0927 13:58:54.979332 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:54.979422 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:39555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 13:58:54.990236 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 3410
I20260927 13:58:55.005978 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:54.991100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.22.26.195:35509)
0927 13:58:54.991269 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:54.991273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:54.991336 (+ 63us) server_negotiation.cc:407] Connection header received
0927 13:58:54.991517 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:54.991521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:54.991590 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:54.991674 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:54.993424 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.994176 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.996118 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:54.996508 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:54.996609 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:54.999842 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:54.999857 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:54.999873 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:54.999915 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:55.003001 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:55.003785 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:55.003796 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:55.003800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:55.003991 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:55.004361 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:55.004367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:55.004371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:55.004832 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:55.005145 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:55.005379 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 13:58:55.005607 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:58:55.006654 3542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:58:55.059960 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:55.059499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:39555 (local address 127.0.0.1:55738)
0927 13:58:55.059708 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:55.059795 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:39555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 13:58:55.061036 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:55.060670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:39555 (local address 127.0.0.1:55752)
0927 13:58:55.060864 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:55.060931 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:39555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 13:58:55.098443 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 3410
I20260927 13:58:55.115283 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 3030
I20260927 13:58:55.116567 3093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:58:55.229099 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 3030
I20260927 13:58:55.251466 22635 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 13:58:55.586392 22635 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 13:58:55.586621 22635 mini_ranger.cc:66] Stopping Ranger...
I20260927 13:58:55.932031 22635 mini_ranger.cc:68] Stopped Ranger
2026-09-27 13:58:55.932 UTC [2082] LOG: received smart shutdown request
2026-09-27 13:58:55.935 UTC [2082] LOG: background worker "logical replication launcher" (PID 2091) exited with exit code 1
2026-09-27 13:58:55.935 UTC [2086] LOG: shutting down
2026-09-27 13:58:56.112 UTC [2082] LOG: database system is shut down
2026-09-27T13:58:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52535 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:58:56 dist-test-slave-gkks krb5kdc[3608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:58:56 dist-test-slave-gkks krb5kdc[3608](info): set up 2 sockets
Sep 27 13:58:56 dist-test-slave-gkks krb5kdc[3608](info): commencing operation
krb5kdc: starting...
W20260927 13:58:57.395838 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:58:57 dist-test-slave-gkks krb5kdc[3608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:58:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:58:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:58:57.637099 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:35789
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:35789,127.22.26.253:43909,127.22.26.252:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:58:57.969138 3621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:57.969614 3621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:57.969771 3621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:57.978322 3621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:58:57.978428 3621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:57.978483 3621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:57.978542 3621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:58:57.978585 3621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:58:57.988829 3621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:35789,127.22.26.253:43909,127.22.26.252:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:35789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:57.990217 3621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:57.991631 3621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:58.003423 3629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:58.003636 3626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:58.003437 3627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:58.004433 3621 server_base.cc:1034] running on GCE node
I20260927 13:58:58.005270 3621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:58.006695 3621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:58.007907 3621 hybrid_clock.cc:648] HybridClock initialized: now 1790517538007853 us; error 46 us; skew 500 ppm
I20260927 13:58:58.008322 3621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:58 dist-test-slave-gkks krb5kdc[3608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517538, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:58.018658 3621 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:58:58.020555 3621 webserver.cc:460] Webserver started at http://127.22.26.254:40911/ using document root <none> and password file <none>
I20260927 13:58:58.021358 3621 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:58.021523 3621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:58:58.021872 3621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:58:58.025280 3621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "eaf3934199e749378cea15febc2024ab"
format_stamp: "Formatted at 2026-09-27 13:58:58 on dist-test-slave-gkks"
server_key: "deaf777705a638ff1c851882a6630d87"
server_key_iv: "08115ea8e4661e108ef5c0948b97315d"
server_key_version: "encryptionkey@0"
I20260927 13:58:58.026182 3621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "eaf3934199e749378cea15febc2024ab"
format_stamp: "Formatted at 2026-09-27 13:58:58 on dist-test-slave-gkks"
server_key: "deaf777705a638ff1c851882a6630d87"
server_key_iv: "08115ea8e4661e108ef5c0948b97315d"
server_key_version: "encryptionkey@0"
I20260927 13:58:58.031890 3621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 13:58:58.035501 3636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:58.036973 3621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:58:58.037194 3621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "eaf3934199e749378cea15febc2024ab"
format_stamp: "Formatted at 2026-09-27 13:58:58 on dist-test-slave-gkks"
server_key: "deaf777705a638ff1c851882a6630d87"
server_key_iv: "08115ea8e4661e108ef5c0948b97315d"
server_key_version: "encryptionkey@0"
I20260927 13:58:58.037416 3621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:58:58.079367 3621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:58.084074 3621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:58.084385 3621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:58.104664 3621 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:35789
I20260927 13:58:58.104713 3687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:35789 every 8 connection(s)
I20260927 13:58:58.106089 3621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:58:58.110220 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 3621
I20260927 13:58:58.110483 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:58:58.110587 3688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:58:58.111475 22635 external_mini_cluster.cc:1428] Setting key f4855d5d2f8c12d536af32a88c4927ad
I20260927 13:58:58.121950 3688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.26.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.253@KRBTEST.COM" created.
Sep 27 13:58:58 dist-test-slave-gkks krb5kdc[3608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517538, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:58:58.149945 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.125627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.22.26.254:35789)
0927 13:58:58.126595 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:58.126618 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:58.127112 (+ 494us) server_negotiation.cc:407] Connection header received
0927 13:58:58.129346 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:58.129352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:58.129591 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:58.129832 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:58.131239 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.132685 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.134216 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.134828 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.135070 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:58.140261 (+ 5191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:58.140277 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:58.140298 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:58.140355 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:58.145005 (+ 4650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:58.145893 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:58.145906 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:58.145913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:58.146141 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:58.146573 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:58.146581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:58.146588 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:58.147378 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:58.147565 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:58.148383 (+ 818us) server_negotiation.cc:299] Negotiation successful
0927 13:58:58.148729 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":672,"thread_start_us":438,"threads_started":1}
I20260927 13:58:58.150002 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.125638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35789 (local address 127.0.0.1:36702)
0927 13:58:58.126919 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.126955 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:58.128928 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:58.129832 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:58.129840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:58.130632 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:58.130984 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:58.131003 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.132860 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.132866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:58.134066 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:58.134080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.135118 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.135126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:58.135317 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:58.136054 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:58.136090 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:58.139974 (+ 3884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:58.145210 (+ 5236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:58.145233 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:58.145243 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:58.145719 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:58.146266 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:58.146270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:58.146273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:58.146442 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:58.147548 (+ 1106us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:58.147557 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:58.147882 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 13:58:58.148221 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 13:58:58.148930 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":114,"spinlock_wait_cycles":26368,"thread_start_us":825,"threads_started":1}
I20260927 13:58:58.153898 3688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:58:58.155236 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.154722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43909 (local address 127.0.0.1:32820)
0927 13:58:58.154894 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.155008 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 13:58:58.155539 3640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
I20260927 13:58:58.157286 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.156909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43909 (local address 127.0.0.1:32832)
0927 13:58:58.157082 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.157153 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 13:58:58.157521 3688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:43909: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.253.keytab.
Entry for principal kudu/127.22.26.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.253.keytab.
I20260927 13:58:58.171648 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.253:43909
--webserver_interface=127.22.26.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.253.keytab
--principal=kudu/127.22.26.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:35789,127.22.26.253:43909,127.22.26.252:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:58:58.206861 3688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } attempt: 1
I20260927 13:58:58.208030 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.207582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43909 (local address 127.0.0.1:32838)
0927 13:58:58.207777 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.207902 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 13:58:58.208997 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.208658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43909 (local address 127.0.0.1:32846)
0927 13:58:58.208801 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.208875 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 13:58:58.209177 3688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:43909: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
I20260927 13:58:58.277464 3688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } attempt: 2
I20260927 13:58:58.278723 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.278201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43909 (local address 127.0.0.1:32858)
0927 13:58:58.278485 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.278585 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 13:58:58.279788 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.279427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43909 (local address 127.0.0.1:32868)
0927 13:58:58.279591 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.279663 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 13:58:58.279983 3688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:43909: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
I20260927 13:58:58.371234 3688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } attempt: 3
I20260927 13:58:58.372399 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.371937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43909 (local address 127.0.0.1:32872)
0927 13:58:58.372151 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.372248 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 13:58:58.373417 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.373053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43909 (local address 127.0.0.1:32888)
0927 13:58:58.373198 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.373285 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 13:58:58.373608 3688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:43909: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
W20260927 13:58:58.503261 3695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:58.503595 3695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:58.503754 3695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:58.511004 3695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:58:58.511185 3695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:58.511289 3695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:58.511389 3695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:58:58.511461 3695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:58:58.516853 3688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } attempt: 4
I20260927 13:58:58.518064 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.517489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43909 (local address 127.0.0.1:32896)
0927 13:58:58.517822 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.517922 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260927 13:58:58.519307 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.518866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43909 (local address 127.0.0.1:32906)
0927 13:58:58.519018 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.519163 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 13:58:58.519520 3688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:43909: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43909: connect: Connection refused (error 111)
I20260927 13:58:58.521847 3695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:35789,127.22.26.253:43909,127.22.26.252:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.253.keytab
--principal=kudu/127.22.26.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.253:43909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.26.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:58.523358 3695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:58.524681 3695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:58.537554 3700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:58.537482 3703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:58.537595 3701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:58.538441 3695 server_base.cc:1034] running on GCE node
I20260927 13:58:58.539312 3695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:58.540736 3695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:58.541955 3695 hybrid_clock.cc:648] HybridClock initialized: now 1790517538541885 us; error 64 us; skew 500 ppm
I20260927 13:58:58.542299 3695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:58 dist-test-slave-gkks krb5kdc[3608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517538, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:58.545433 3695 init.cc:377] Logged in from keytab as kudu/127.22.26.253@KRBTEST.COM (short username kudu)
I20260927 13:58:58.546941 3695 webserver.cc:460] Webserver started at http://127.22.26.253:41541/ using document root <none> and password file <none>
I20260927 13:58:58.547686 3695 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:58.547827 3695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:58:58.548144 3695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:58:58.551034 3695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/data/instance:
uuid: "82773dabb86f48c6bdc7f59888938fa5"
format_stamp: "Formatted at 2026-09-27 13:58:58 on dist-test-slave-gkks"
server_key: "2bb300f2bbfed3f0d4873955065352d5"
server_key_iv: "c600e32d1c2ff3c7778f973d055b02de"
server_key_version: "encryptionkey@0"
I20260927 13:58:58.551892 3695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/wal/instance:
uuid: "82773dabb86f48c6bdc7f59888938fa5"
format_stamp: "Formatted at 2026-09-27 13:58:58 on dist-test-slave-gkks"
server_key: "2bb300f2bbfed3f0d4873955065352d5"
server_key_iv: "c600e32d1c2ff3c7778f973d055b02de"
server_key_version: "encryptionkey@0"
I20260927 13:58:58.557010 3695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:58:58.560385 3710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:58.561652 3695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:58:58.561825 3695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/wal
uuid: "82773dabb86f48c6bdc7f59888938fa5"
format_stamp: "Formatted at 2026-09-27 13:58:58 on dist-test-slave-gkks"
server_key: "2bb300f2bbfed3f0d4873955065352d5"
server_key_iv: "c600e32d1c2ff3c7778f973d055b02de"
server_key_version: "encryptionkey@0"
I20260927 13:58:58.562024 3695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:58:58.572896 3695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:58.576929 3695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:58.577220 3695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:58.595767 3695 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.253:43909
I20260927 13:58:58.595808 3761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.253:43909 every 8 connection(s)
I20260927 13:58:58.597149 3695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/data/info.pb
I20260927 13:58:58.600872 3762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:58:58.604653 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 3695
I20260927 13:58:58.604912 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-1/wal/instance
I20260927 13:58:58.605386 22635 external_mini_cluster.cc:1428] Setting key 01992ad891d4f9dafead137f2c7978ff
I20260927 13:58:58.607439 3762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:58:58 dist-test-slave-gkks krb5kdc[3608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517538, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.253@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.26.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.252@KRBTEST.COM" created.
I20260927 13:58:58.628710 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.610125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.22.26.254:35789)
0927 13:58:58.610300 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:58.610304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:58.611721 (+ 1417us) server_negotiation.cc:407] Connection header received
0927 13:58:58.614225 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:58.614231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:58.614325 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:58.614441 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:58.616075 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.616739 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.618288 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.618749 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.618902 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:58.622491 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:58.622496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:58.622499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:58.622533 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:58.625227 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:58.626065 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:58.626069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:58.626072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:58.626191 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:58.626727 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:58.626731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:58.626733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:58.627020 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:58.627213 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:58.628156 (+ 943us) server_negotiation.cc:299] Negotiation successful
0927 13:58:58.628387 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:58:58.629750 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.610715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35789 (local address 127.0.0.1:36712)
0927 13:58:58.611559 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.611602 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:58.613951 (+ 2349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:58.614681 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:58.614690 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:58.615308 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:58.615875 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:58.615892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.616909 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.616913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:58.618139 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:58.618151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.619008 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.619016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:58.619290 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:58.619986 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:58.620008 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:58.622243 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:58.625375 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:58.625384 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:58.625400 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:58.625917 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:58.626309 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:58.626314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:58.626318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:58.626594 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:58.627187 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:58.627195 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:58.627653 (+ 458us) client_negotiation.cc:769] Sending connection context
0927 13:58:58.628020 (+ 367us) client_negotiation.cc:240] Negotiation successful
0927 13:58:58.628545 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":409,"threads_started":1}
I20260927 13:58:58.631701 3762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:58:58 dist-test-slave-gkks krb5kdc[3608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517538, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.253@KRBTEST.COM for kudu/127.22.26.253@KRBTEST.COM
I20260927 13:58:58.649585 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.632571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43909 (local address 127.0.0.1:32910)
0927 13:58:58.632783 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.632807 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:58.632917 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:58.633946 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:58.633950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:58.634303 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:58.634437 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:58.634444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.636301 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.636307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:58.637828 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:58.637846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.638763 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.638771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:58.639019 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:58.639717 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:58.639744 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:58.642441 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:58.646894 (+ 4453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:58.646900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:58.646905 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:58.647361 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:58.647821 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:58.647825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:58.647827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:58.647966 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:58.648803 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:58.648807 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:58.648938 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 13:58:58.649070 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 13:58:58.649291 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 13:58:58.650038 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.632720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.22.26.253:43909)
0927 13:58:58.633442 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:58.633452 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:58.633484 (+ 32us) server_negotiation.cc:407] Connection header received
0927 13:58:58.633547 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:58.633557 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:58.633789 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:58.633949 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:58.634597 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.636158 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.637990 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.638602 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.638709 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:58.642674 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:58.642690 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:58.642711 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:58.642755 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:58.646748 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:58.647519 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:58.647530 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:58.647535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:58.647693 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:58.648109 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:58.648117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:58.648121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:58.648640 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:58.648881 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:58.649288 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 13:58:58.649600 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":380,"threads_started":1}
I20260927 13:58:58.652474 3762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:58:58.653497 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.653092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49818)
0927 13:58:58.653238 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.653348 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 13:58:58.653755 3714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
I20260927 13:58:58.655500 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.655066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49824)
0927 13:58:58.655276 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.655387 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 13:58:58.655752 3762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33377: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.252.keytab.
Entry for principal kudu/127.22.26.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.252.keytab.
I20260927 13:58:58.662038 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.252:33377
--webserver_interface=127.22.26.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.252.keytab
--principal=kudu/127.22.26.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:35789,127.22.26.253:43909,127.22.26.252:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:58:58.705108 3762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } attempt: 1
I20260927 13:58:58.706365 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.705894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49828)
0927 13:58:58.706107 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.706213 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 13:58:58.707383 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.707025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49836)
0927 13:58:58.707211 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.707282 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 13:58:58.707584 3762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33377: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
I20260927 13:58:58.775820 3762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } attempt: 2
I20260927 13:58:58.776989 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.776524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49844)
0927 13:58:58.776731 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.776830 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 13:58:58.777998 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.777666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49858)
0927 13:58:58.777818 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.777890 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 13:58:58.778189 3762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33377: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
I20260927 13:58:58.818796 3688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } attempt: 5
Sep 27 13:58:58 dist-test-slave-gkks krb5kdc[3608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517538, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for kudu/127.22.26.253@KRBTEST.COM
I20260927 13:58:58.834699 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.819589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43909 (local address 127.0.0.1:32924)
0927 13:58:58.819867 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.819895 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:58.820081 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:58.820503 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:58.820509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:58.821130 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:58.821431 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:58.821449 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.822486 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.822494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:58.824249 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:58.824269 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.825086 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.825093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:58.825270 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:58.825951 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:58.825981 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:58.828578 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:58.832373 (+ 3795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:58.832377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:58.832380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:58.832778 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:58.833197 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:58.833200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:58.833203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:58.833324 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:58.833927 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:58.833931 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:58.834071 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 13:58:58.834185 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 13:58:58.834412 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 13:58:58.834862 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.819723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.22.26.253:43909)
0927 13:58:58.819901 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:58.819905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:58.819988 (+ 83us) server_negotiation.cc:407] Connection header received
0927 13:58:58.820230 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:58.820235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:58.820316 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:58.820403 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:58.821638 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.822314 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.824438 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:58.824907 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:58.825015 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:58.828811 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:58.828819 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:58.828824 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:58.828868 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:58.832209 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:58.832907 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:58.832913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:58.832916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:58.833082 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:58.833428 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:58.833433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:58.833436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:58.833804 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:58.833955 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:58.834317 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 13:58:58.834558 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:58:58.835983 3688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:58:58.837066 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.836652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49864)
0927 13:58:58.836843 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.836924 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 13:58:58.838042 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.837698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49880)
0927 13:58:58.837867 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.837945 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 13:58:58.838279 3688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33377: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
I20260927 13:58:58.869473 3762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } attempt: 3
I20260927 13:58:58.870814 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.870217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49886)
0927 13:58:58.870490 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.870612 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 13:58:58.872237 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.871641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49890)
0927 13:58:58.871924 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.872053 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260927 13:58:58.872529 3762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33377: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
I20260927 13:58:58.889541 3688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } attempt: 1
I20260927 13:58:58.890910 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.890239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49896)
0927 13:58:58.890524 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.890666 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 13:58:58.892221 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.891763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49908)
0927 13:58:58.891950 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.892064 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 13:58:58.892542 3688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33377: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
I20260927 13:58:58.960791 3688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } attempt: 2
I20260927 13:58:58.962157 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.961507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49918)
0927 13:58:58.961781 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.961905 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 13:58:58.963374 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:58.962962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49926)
0927 13:58:58.963143 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:58.963237 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 13:58:58.963608 3688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33377: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
W20260927 13:58:59.004283 3769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:58:59.004587 3769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:58:59.004678 3769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:58:59.011977 3769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:58:59.012082 3769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:58:59.012130 3769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:58:59.012200 3769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:58:59.012246 3769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:58:59.015780 3762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } attempt: 4
I20260927 13:58:59.017025 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.016458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49940)
0927 13:58:59.016716 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:59.016825 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 13:58:59.018112 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.017739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49944)
0927 13:58:59.017890 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:59.017987 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 13:58:59.018396 3762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33377: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
I20260927 13:58:59.022943 3769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:35789,127.22.26.253:43909,127.22.26.252:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.252.keytab
--principal=kudu/127.22.26.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.252:33377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.26.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:58:59.024421 3769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:58:59.025673 3769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:58:59.037575 3775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:59.037542 3777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:58:59.037597 3774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:58:59.038266 3769 server_base.cc:1034] running on GCE node
I20260927 13:58:59.039151 3769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:58:59.040608 3769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:58:59.041826 3769 hybrid_clock.cc:648] HybridClock initialized: now 1790517539041769 us; error 51 us; skew 500 ppm
I20260927 13:58:59.042284 3769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:58:59 dist-test-slave-gkks krb5kdc[3608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517539, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:58:59.045540 3769 init.cc:377] Logged in from keytab as kudu/127.22.26.252@KRBTEST.COM (short username kudu)
I20260927 13:58:59.047201 3769 webserver.cc:460] Webserver started at http://127.22.26.252:34853/ using document root <none> and password file <none>
I20260927 13:58:59.047899 3769 fs_manager.cc:359] Metadata directory not provided
I20260927 13:58:59.047998 3769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:58:59.048262 3769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:58:59.051266 3769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/data/instance:
uuid: "4c509d80eeaa49a9a15df7e2155cc33a"
format_stamp: "Formatted at 2026-09-27 13:58:59 on dist-test-slave-gkks"
server_key: "f0e88bf453488f6613f6ba3d696cb6a7"
server_key_iv: "4eb2e4d10382159cdc3b0b6874b9b0f4"
server_key_version: "encryptionkey@0"
I20260927 13:58:59.052032 3769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/wal/instance:
uuid: "4c509d80eeaa49a9a15df7e2155cc33a"
format_stamp: "Formatted at 2026-09-27 13:58:59 on dist-test-slave-gkks"
server_key: "f0e88bf453488f6613f6ba3d696cb6a7"
server_key_iv: "4eb2e4d10382159cdc3b0b6874b9b0f4"
server_key_version: "encryptionkey@0"
I20260927 13:58:59.057181 3769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:58:59.060652 3784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:58:59.062016 3769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:58:59.062191 3769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/wal
uuid: "4c509d80eeaa49a9a15df7e2155cc33a"
format_stamp: "Formatted at 2026-09-27 13:58:59 on dist-test-slave-gkks"
server_key: "f0e88bf453488f6613f6ba3d696cb6a7"
server_key_iv: "4eb2e4d10382159cdc3b0b6874b9b0f4"
server_key_version: "encryptionkey@0"
I20260927 13:58:59.062417 3769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:58:59.069815 3688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } attempt: 3
I20260927 13:58:59.070914 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.070488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49956)
0927 13:58:59.070667 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:59.070758 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 13:58:59.071887 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.071599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49960)
0927 13:58:59.071723 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:59.071787 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 13:58:59.072136 3688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33377: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33377: connect: Connection refused (error 111)
I20260927 13:58:59.097250 3769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:58:59.101657 3769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:58:59.101966 3769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:58:59.121841 3769 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.252:33377
I20260927 13:58:59.121876 3835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.252:33377 every 8 connection(s)
I20260927 13:58:59.123338 3769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/data/info.pb
I20260927 13:58:59.125268 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 3769
I20260927 13:58:59.125466 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790517330258165-22635-0/minicluster-data/master-2/wal/instance
I20260927 13:58:59.125907 22635 external_mini_cluster.cc:1428] Setting key dac2a1de7962a54c39dc901743469c8d
I20260927 13:58:59.127722 3836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:58:59.136643 3836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:58:59 dist-test-slave-gkks krb5kdc[3608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.252@KRBTEST.COM
Sep 27 13:58:59 dist-test-slave-gkks krb5kdc[3608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517539, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.252@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:58:59.150301 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.127765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.22.26.252:33377)
0927 13:58:59.128640 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:59.128651 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:59.128691 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:58:59.130274 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:59.130293 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:59.130691 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:59.131012 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:59.132936 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.134285 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.136223 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.136905 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.137031 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.142101 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:59.142111 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:59.142126 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:59.142173 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:59.145824 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.146713 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.146722 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.146734 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.146929 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.147407 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.147417 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.147422 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.148139 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:59.148477 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:59.148894 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 13:58:59.149311 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":381,"threads_started":1}
I20260927 13:58:59.156157 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.139714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:35789 (local address 127.0.0.1:36718)
0927 13:58:59.140660 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:59.140711 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:59.140889 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:59.141377 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:59.141385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:59.142160 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:59.142568 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:59.142582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.143737 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.143745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:59.145496 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:59.145515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.146397 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.146402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:59.146524 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.147123 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:59.147144 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:59.149721 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:59.153202 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:59.153213 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:59.153220 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:59.153857 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:59.154264 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:59.154268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:59.154270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:59.154397 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:59.154964 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:59.154971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:59.155221 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 13:58:59.155402 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 13:58:59.155723 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":537,"threads_started":1}
I20260927 13:58:59.156186 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.139701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.22.26.254:35789)
0927 13:58:59.140313 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:59.140319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:59.140822 (+ 503us) server_negotiation.cc:407] Connection header received
0927 13:58:59.141044 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:59.141049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:59.141119 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:59.141309 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:59.142730 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.143577 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.145654 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.146223 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.146416 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.149920 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:59.149926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:59.149929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:59.149960 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:59.153063 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.153969 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.153979 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.153991 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.154151 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.154518 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.154547 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.154550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.154822 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:59.154979 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:59.155513 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 13:58:59.155730 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":276,"threads_started":1}
I20260927 13:58:59.157485 3836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:58:59 dist-test-slave-gkks krb5kdc[3608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
Sep 27 13:58:59 dist-test-slave-gkks krb5kdc[3608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517539, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.252@KRBTEST.COM for kudu/127.22.26.253@KRBTEST.COM
I20260927 13:58:59.169098 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.153541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.22.26.254:35789)
0927 13:58:59.154164 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:59.154169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:59.154192 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:58:59.154239 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:59.154243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:59.154307 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:59.154419 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:59.155774 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.156896 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.158437 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.158862 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.159011 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.163049 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:59.163060 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:59.163067 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:59.163144 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:59.166176 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.166945 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.166951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.166955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.167133 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.167540 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.167547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.167553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.167848 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:59.168102 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:59.168532 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 13:58:59.168716 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":256,"threads_started":1}
I20260927 13:58:59.172056 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.158277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43909 (local address 127.0.0.1:32936)
0927 13:58:59.158562 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:59.158577 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:59.158707 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:59.159035 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:59.159040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:59.159443 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:59.159583 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:59.159592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.160558 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.160564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:59.161406 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:59.161417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.162129 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.162135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:59.162338 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.162921 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:59.162946 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:59.165383 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:59.169164 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:59.169169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:59.169174 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:59.169698 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:59.170164 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:59.170169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:59.170172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:59.170344 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:59.170948 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:59.170954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:59.171179 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 13:58:59.171372 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 13:58:59.171689 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260927 13:58:59.172103 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.158279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.22.26.253:43909)
0927 13:58:59.158451 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:59.158455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:59.158653 (+ 198us) server_negotiation.cc:407] Connection header received
0927 13:58:59.158843 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:59.158847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:59.158910 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:59.159015 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:59.159744 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.160411 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.161549 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.161968 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.162114 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.165625 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:59.165631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:59.165634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:59.165669 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:59.169013 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.169843 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.169847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.169849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.169985 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.170462 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.170466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.170469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.170781 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:59.170960 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:59.171491 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 13:58:59.171708 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:58:59.173141 3836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:58:59 dist-test-slave-gkks krb5kdc[3608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.253@KRBTEST.COM
Sep 27 13:58:59 dist-test-slave-gkks krb5kdc[3608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517539, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.252@KRBTEST.COM for kudu/127.22.26.252@KRBTEST.COM
I20260927 13:58:59.188525 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.174057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:49982)
0927 13:58:59.174311 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:59.174330 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:59.174424 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:59.174978 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:59.174986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:59.175647 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:59.175880 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:59.175896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.177250 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.177257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:59.179152 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:59.179171 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.180004 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.180007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:59.180132 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.180687 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:59.180703 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:59.182933 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:59.185901 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:59.185909 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:59.185913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:59.186267 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:59.186658 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:59.186665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:59.186670 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:59.186793 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:59.187360 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:59.187364 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:59.187508 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 13:58:59.187650 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 13:58:59.188015 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":39}
I20260927 13:58:59.188665 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.171021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.22.26.253:43909)
0927 13:58:59.172704 (+ 1683us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:59.172710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:59.172873 (+ 163us) server_negotiation.cc:407] Connection header received
0927 13:58:59.172928 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:59.172932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:59.173019 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:59.173136 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:59.174770 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.175605 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.177390 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.177834 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.177987 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.182880 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:59.182885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:59.182888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:59.182923 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:59.185847 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.186655 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.186661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.186665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.186808 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.187200 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.187206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.187209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.187469 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:59.187765 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:59.188034 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 13:58:59.188269 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1592,"thread_start_us":8187,"threads_started":1}
I20260927 13:58:59.188551 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.174159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.22.26.252:33377)
0927 13:58:59.174367 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:59.174376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:59.174400 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:58:59.174530 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:59.174535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:59.174636 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:59.174781 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:59.176065 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.177099 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.179329 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.179767 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.179838 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.183151 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:59.183157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:59.183161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:59.183198 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:59.185754 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.186385 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.186392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.186396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.186549 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.186896 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.186903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.186906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.187252 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:59.187400 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:59.187733 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 13:58:59.188204 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":68}
I20260927 13:58:59.193856 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:58:59.194916 3836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a: Bootstrap starting.
I20260927 13:58:59.199297 3836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a: Neither blocks nor log segments found. Creating new log.
I20260927 13:58:59.200698 3836 log.cc:824] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a: Log is configured to *not* fsync() on all Append() calls
I20260927 13:58:59.210289 3836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a: No bootstrap required, opened a new log
I20260927 13:58:59.221007 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.199996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.22.26.253:43909)
0927 13:58:59.200172 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:59.200177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:59.201839 (+ 1662us) server_negotiation.cc:407] Connection header received
0927 13:58:59.202172 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:59.202180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:59.202278 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:59.202363 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:59.205279 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.206741 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.209025 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.209664 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.209768 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.213641 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:59.213648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:59.213653 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:59.213695 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:59.217222 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.218304 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.218312 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.218316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.218507 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.219004 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.219009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.219012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.219386 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:59.219629 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:59.220425 (+ 796us) server_negotiation.cc:299] Negotiation successful
0927 13:58:59.220664 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 13:58:59.221385 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.200165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.22.26.254:35789)
0927 13:58:59.200414 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:59.200423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:59.201098 (+ 675us) server_negotiation.cc:407] Connection header received
0927 13:58:59.201258 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:59.201266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:59.201392 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:59.201503 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:59.204300 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.205696 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.208646 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.209318 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.209416 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.213582 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:59.213589 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:59.213594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:59.213640 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:59.217821 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.218897 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.218905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.218910 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.219095 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.219569 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.219575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.219579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.219944 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:59.220185 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:59.220759 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 13:58:59.221008 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:58:59.225242 3836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } }
I20260927 13:58:59.225693 3836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:58:59.225858 3836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c509d80eeaa49a9a15df7e2155cc33a, State: Initialized, Role: FOLLOWER
I20260927 13:58:59.226565 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.201838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.22.26.252:33377)
0927 13:58:59.202043 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:59.202050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:59.208873 (+ 6823us) server_negotiation.cc:407] Connection header received
0927 13:58:59.209261 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:59.209268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:59.209342 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:59.209413 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:59.212209 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.213362 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.215360 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.216189 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.216290 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.219951 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:59.219958 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:59.219962 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:59.220004 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:59.223730 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.224404 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.224409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.224414 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.224601 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.224982 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.224989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.224994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.225389 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:59.225661 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:59.225999 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 13:58:59.226213 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:58:59.226568 3836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } }
I20260927 13:58:59.227586 3852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } } }
I20260927 13:58:59.228055 3852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:58:59.228920 3836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [sys.catalog]: configured and running, proceeding with master startup.
W20260927 13:58:59.246975 3863 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:58:59.247275 3863 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:58:59.249363 3688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } attempt: 4
I20260927 13:58:59.242290 3769 master_runner.cc:186] Error getting master registration for 127.22.26.254:35789: OK, instance_id { permanent_uuid: "eaf3934199e749378cea15febc2024ab" instance_seqno: 1790517538098587 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:58:59.255983 3769 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 13:58:59 dist-test-slave-gkks krb5kdc[3608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517538, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for kudu/127.22.26.252@KRBTEST.COM
I20260927 13:58:59.272042 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.249955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:50008)
0927 13:58:59.250160 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:59.250180 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:59.250303 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:59.250851 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:59.250856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:59.251383 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:59.251595 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:59.251606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.254604 (+ 2998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.254610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:59.256057 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:59.256073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.257110 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.257116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:59.257317 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.258059 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:59.258081 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:59.261297 (+ 3216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:59.268611 (+ 7314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:59.268619 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:59.268625 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:59.269249 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:59.269794 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:59.269801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:59.269806 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:59.270010 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:59.270839 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:59.270846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:59.271040 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 13:58:59.271221 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 13:58:59.271575 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 13:58:59.273460 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.250035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.22.26.252:33377)
0927 13:58:59.250195 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:59.250203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:59.250227 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:58:59.250448 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:59.250452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:59.250559 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:59.250649 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:59.251766 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.252971 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.256195 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.256925 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.257065 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.261556 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:59.261564 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:59.261571 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:59.261614 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:59.268400 (+ 6786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.269418 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.269426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.269431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.269640 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.270162 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.270169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.270174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.270607 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:59.270769 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:59.272709 (+ 1940us) server_negotiation.cc:299] Negotiation successful
0927 13:58:59.272999 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:58:59.280550 3688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab: Bootstrap starting.
I20260927 13:58:59.285287 3688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab: Neither blocks nor log segments found. Creating new log.
I20260927 13:58:59.286880 3688 log.cc:824] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab: Log is configured to *not* fsync() on all Append() calls
I20260927 13:58:59.290777 3688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab: No bootstrap required, opened a new log
I20260927 13:58:59.303786 3688 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } }
I20260927 13:58:59.304299 3688 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:58:59.304458 3688 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf3934199e749378cea15febc2024ab, State: Initialized, Role: FOLLOWER
I20260927 13:58:59.305213 3688 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } }
I20260927 13:58:59.306377 3866 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } } }
I20260927 13:58:59.306859 3866 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:58:59.307925 3688 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:58:59.317610 3762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } attempt: 5
W20260927 13:58:59.322232 3877 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:58:59.322449 3877 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 13:58:59 dist-test-slave-gkks krb5kdc[3608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517538, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.253@KRBTEST.COM for kudu/127.22.26.252@KRBTEST.COM
W20260927 13:58:59.337524 3642 tablet.cc:2369] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:58:59.341902 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.318970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.22.26.252:33377)
0927 13:58:59.319592 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0927 13:58:59.319600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:58:59.319624 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:58:59.319690 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:58:59.319698 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:58:59.319785 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:58:59.319902 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:58:59.321692 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.322819 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.325007 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.325741 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.325881 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.330268 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:58:59.330274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:58:59.330280 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:58:59.330321 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:58:59.336575 (+ 6254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.338701 (+ 2126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.338706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.338713 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.338878 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:58:59.339356 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:58:59.339361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:58:59.339364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:58:59.339735 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:58:59.339897 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 13:58:59.341179 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0927 13:58:59.341421 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464}
I20260927 13:58:59.342281 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:58:59.318208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33377 (local address 127.0.0.1:50010)
0927 13:58:59.318634 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 13:58:59.318655 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:58:59.318768 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:58:59.320478 (+ 1710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:58:59.320484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:58:59.321025 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:58:59.321253 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:59.321266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.323370 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.323377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:59.324822 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:58:59.324839 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:58:59.326111 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:58:59.326118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:58:59.326328 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:58:59.327020 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:58:59.327046 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:58:59.330050 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:58:59.336739 (+ 6689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:59.336745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:59.336749 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:59.338552 (+ 1803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:59.339017 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:58:59.339022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:58:59.339026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:58:59.339229 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:58:59.339962 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:58:59.339967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:58:59.340130 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 13:58:59.340281 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 13:58:59.340602 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285}
I20260927 13:58:59.348724 3762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5: Bootstrap starting.
I20260927 13:58:59.333065 3621 master_runner.cc:186] Error getting master registration for 127.22.26.253:43909: OK, instance_id { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" instance_seqno: 1790517538590561 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:58:59.349447 3621 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 13:58:59.354271 3790 tablet.cc:2369] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:58:59.355253 3762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5: Neither blocks nor log segments found. Creating new log.
I20260927 13:58:59.356600 3762 log.cc:824] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:58:59.360096 3762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5: No bootstrap required, opened a new log
I20260927 13:58:59.372390 3762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } }
I20260927 13:58:59.372828 3762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:58:59.372951 3762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82773dabb86f48c6bdc7f59888938fa5, State: Initialized, Role: FOLLOWER
I20260927 13:58:59.373663 3762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } }
I20260927 13:58:59.374686 3880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } } }
I20260927 13:58:59.375157 3880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:58:59.376047 3762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:58:59.394501 3695 master_runner.cc:263] No leader master found from master 82773dabb86f48c6bdc7f59888938fa5
I20260927 13:58:59.394657 3695 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 13:58:59.397558 3891 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:58:59.397789 3891 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:58:59.434731 3852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:58:59.435068 3852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } }
I20260927 13:58:59.436818 3852 leader_election.cc:288] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf3934199e749378cea15febc2024ab (127.22.26.254:35789), 82773dabb86f48c6bdc7f59888938fa5 (127.22.26.253:43909)
I20260927 13:58:59.437543 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf3934199e749378cea15febc2024ab" is_pre_election: true
I20260927 13:58:59.437803 3737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82773dabb86f48c6bdc7f59888938fa5" is_pre_election: true
I20260927 13:58:59.438156 3663 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c509d80eeaa49a9a15df7e2155cc33a in term 0.
I20260927 13:58:59.438383 3737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c509d80eeaa49a9a15df7e2155cc33a in term 0.
I20260927 13:58:59.438800 3786 leader_election.cc:302] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c509d80eeaa49a9a15df7e2155cc33a, eaf3934199e749378cea15febc2024ab; no voters:
I20260927 13:58:59.439568 3852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:58:59.439709 3852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:58:59.439849 3852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:58:59.443517 3852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } }
I20260927 13:58:59.444633 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf3934199e749378cea15febc2024ab"
I20260927 13:58:59.444866 3663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:58:59.448416 3663 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c509d80eeaa49a9a15df7e2155cc33a in term 1.
I20260927 13:58:59.448866 3786 leader_election.cc:302] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c509d80eeaa49a9a15df7e2155cc33a, eaf3934199e749378cea15febc2024ab; no voters:
I20260927 13:58:59.452665 3737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82773dabb86f48c6bdc7f59888938fa5"
I20260927 13:58:59.452899 3737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:58:59.455521 3893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:58:59.456352 3737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c509d80eeaa49a9a15df7e2155cc33a in term 1.
I20260927 13:58:59.457358 3852 leader_election.cc:288] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [CANDIDATE]: Term 1 election: Requested vote from peers eaf3934199e749378cea15febc2024ab (127.22.26.254:35789), 82773dabb86f48c6bdc7f59888938fa5 (127.22.26.253:43909)
I20260927 13:58:59.463346 3893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [term 1 LEADER]: Becoming Leader. State: Replica: 4c509d80eeaa49a9a15df7e2155cc33a, State: Running, Role: LEADER
I20260927 13:58:59.463985 3893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } }
I20260927 13:58:59.485229 3852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c509d80eeaa49a9a15df7e2155cc33a. Latest consensus state: current_term: 1 leader_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } } }
I20260927 13:58:59.485595 3852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [sys.catalog]: This master's current role is: LEADER
I20260927 13:58:59.491714 3896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:58:59.493994 3896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:58:59.510067 3737 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [term 1 FOLLOWER]: Refusing update from remote peer 4c509d80eeaa49a9a15df7e2155cc33a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:58:59.511042 3893 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:58:59.518250 3663 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [term 1 FOLLOWER]: Refusing update from remote peer 4c509d80eeaa49a9a15df7e2155cc33a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:58:59.519186 3893 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:58:59.537717 3880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c509d80eeaa49a9a15df7e2155cc33a. Latest consensus state: current_term: 1 leader_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } } }
I20260927 13:58:59.538053 3880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:58:59.538363 3897 mvcc.cc:205] Tried to move back new op lower bound from 7333959841817939968 to 7333959841651105792. Current Snapshot: MvccSnapshot[applied={T|T < 7333959841817939968}]
I20260927 13:58:59.540392 3880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } } }
I20260927 13:58:59.540625 3880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:58:59.542186 3866 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c509d80eeaa49a9a15df7e2155cc33a. Latest consensus state: current_term: 1 leader_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } } }
I20260927 13:58:59.542553 3866 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:58:59.551210 3899 mvcc.cc:205] Tried to move back new op lower bound from 7333959841817939968 to 7333959841651105792. Current Snapshot: MvccSnapshot[applied={T|T < 7333959841817939968}]
I20260927 13:58:59.551674 3893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } } }
I20260927 13:58:59.552007 3893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [sys.catalog]: This master's current role is: LEADER
I20260927 13:58:59.552285 3893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } } }
I20260927 13:58:59.552539 3893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a [sys.catalog]: This master's current role is: LEADER
I20260927 13:58:59.553754 3896 catalog_manager.cc:1347] Generated new cluster ID: 1b57d64ef7af4625a021cfb3b998676f
I20260927 13:58:59.553848 3896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:58:59.566612 3866 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3934199e749378cea15febc2024ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 35789 } } peers { permanent_uuid: "82773dabb86f48c6bdc7f59888938fa5" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43909 } } peers { permanent_uuid: "4c509d80eeaa49a9a15df7e2155cc33a" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33377 } } }
I20260927 13:58:59.566912 3866 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:58:59.584282 3896 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:58:59.586242 3896 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:58:59.603508 3896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c509d80eeaa49a9a15df7e2155cc33a: Generated new TSK 0
I20260927 13:58:59.604662 3896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:59:00.324506 3877 catalog_manager.cc:1259] Loaded cluster ID: 1b57d64ef7af4625a021cfb3b998676f
I20260927 13:59:00.324611 3877 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab: loading cluster ID for follower catalog manager: success
I20260927 13:59:00.327049 3877 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab: acquiring CA information for follower catalog manager: success
I20260927 13:59:00.328831 3877 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eaf3934199e749378cea15febc2024ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:59:00.399835 3891 catalog_manager.cc:1259] Loaded cluster ID: 1b57d64ef7af4625a021cfb3b998676f
I20260927 13:59:00.399940 3891 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5: loading cluster ID for follower catalog manager: success
I20260927 13:59:00.402397 3891 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5: acquiring CA information for follower catalog manager: success
I20260927 13:59:00.404212 3891 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 82773dabb86f48c6bdc7f59888938fa5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:59:00.726063 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 3621
I20260927 13:59:00.727295 3683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:00.836542 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 3621
I20260927 13:59:00.857622 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 3695
I20260927 13:59:00.858762 3757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:00.965863 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 3695
I20260927 13:59:00.987483 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 3769
I20260927 13:59:00.988656 3831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:01.098919 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 3769
2026-09-27T13:59:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4854 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:59:01 dist-test-slave-gkks krb5kdc[3917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:59:01 dist-test-slave-gkks krb5kdc[3917](info): set up 2 sockets
Sep 27 13:59:01 dist-test-slave-gkks krb5kdc[3917](info): commencing operation
krb5kdc: starting...
W20260927 13:59:02.208058 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:59:02 dist-test-slave-gkks krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:59:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:59:02.451326 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42229
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:42229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:02.780797 3930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:02.781080 3930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:02.781172 3930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:02.788365 3930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:59:02.788476 3930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:02.788525 3930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:02.788586 3930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:59:02.788630 3930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:59:02.798491 3930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:42229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:02.799911 3930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:02.801205 3930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:02.812994 3935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:02.813038 3938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:02.812988 3936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:02.814167 3930 server_base.cc:1034] running on GCE node
I20260927 13:59:02.814949 3930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:02.816440 3930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:02.817647 3930 hybrid_clock.cc:648] HybridClock initialized: now 1790517542817608 us; error 28 us; skew 500 ppm
I20260927 13:59:02.818011 3930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:02 dist-test-slave-gkks krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517542, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:02.821223 3930 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:59:02.822739 3930 webserver.cc:460] Webserver started at http://127.22.26.254:41299/ using document root <none> and password file <none>
I20260927 13:59:02.823508 3930 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:02.823655 3930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:02.823957 3930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:59:02.826934 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "f00438db8e1f42dc98ee1b3ea689fee4"
format_stamp: "Formatted at 2026-09-27 13:59:02 on dist-test-slave-gkks"
server_key: "0836cc3c23e553f4f4e7b3b286ada028"
server_key_iv: "45206df77d9eccb9428b7cd45dae3475"
server_key_version: "encryptionkey@0"
I20260927 13:59:02.827760 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "f00438db8e1f42dc98ee1b3ea689fee4"
format_stamp: "Formatted at 2026-09-27 13:59:02 on dist-test-slave-gkks"
server_key: "0836cc3c23e553f4f4e7b3b286ada028"
server_key_iv: "45206df77d9eccb9428b7cd45dae3475"
server_key_version: "encryptionkey@0"
I20260927 13:59:02.833078 3930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:59:02.836526 3945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:02.837868 3930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 13:59:02.838074 3930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "f00438db8e1f42dc98ee1b3ea689fee4"
format_stamp: "Formatted at 2026-09-27 13:59:02 on dist-test-slave-gkks"
server_key: "0836cc3c23e553f4f4e7b3b286ada028"
server_key_iv: "45206df77d9eccb9428b7cd45dae3475"
server_key_version: "encryptionkey@0"
I20260927 13:59:02.838281 3930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:02.855180 3930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:02.859373 3930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:02.859695 3930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:02.879433 3930 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:42229
I20260927 13:59:02.879428 3996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:42229 every 8 connection(s)
I20260927 13:59:02.880914 3930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:59:02.884711 3997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:02.886845 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 3930
I20260927 13:59:02.887153 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:59:02.887949 22635 external_mini_cluster.cc:1428] Setting key 221ce61609cf79dedecd9998ac878a02
I20260927 13:59:02.896853 3997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4: Bootstrap starting.
I20260927 13:59:02.900699 3997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:02.902074 3997 log.cc:824] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4: Log is configured to *not* fsync() on all Append() calls
Sep 27 13:59:02 dist-test-slave-gkks krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:02.905475 3997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4: No bootstrap required, opened a new log
I20260927 13:59:02.913007 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:02.889808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.22.26.254:42229)
0927 13:59:02.890667 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:02.890678 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:02.890719 (+ 41us) server_negotiation.cc:407] Connection header received
0927 13:59:02.892483 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:02.892507 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:02.892896 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:02.893276 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:02.895174 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:02.896492 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:02.898283 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:02.898962 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:02.899184 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:02.904052 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:02.904064 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:02.904082 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:02.904123 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:02.908162 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:02.908974 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:02.908983 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:02.908993 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:02.909213 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:02.909585 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:02.909590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:02.909594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:02.910326 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:02.910627 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:02.911023 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 13:59:02.911573 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":390,"threads_started":1}
I20260927 13:59:02.915869 3997 raft_consensus.cc:348] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00438db8e1f42dc98ee1b3ea689fee4" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42229 } }
I20260927 13:59:02.916378 3997 raft_consensus.cc:374] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:02.916517 3997 raft_consensus.cc:729] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00438db8e1f42dc98ee1b3ea689fee4, State: Initialized, Role: FOLLOWER
I20260927 13:59:02.917208 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:59:02.917150 3997 consensus_queue.cc:260] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00438db8e1f42dc98ee1b3ea689fee4" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42229 } }
I20260927 13:59:02.917430 3997 raft_consensus.cc:388] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:59:02.917577 3997 raft_consensus.cc:482] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:59:02.917764 3997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:02.920220 3997 raft_consensus.cc:504] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00438db8e1f42dc98ee1b3ea689fee4" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42229 } }
I20260927 13:59:02.920766 3997 leader_election.cc:302] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f00438db8e1f42dc98ee1b3ea689fee4; no voters:
I20260927 13:59:02.921303 3997 leader_election.cc:288] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:59:02.921460 4002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:59:02.922158 4002 raft_consensus.cc:686] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [term 1 LEADER]: Becoming Leader. State: Replica: f00438db8e1f42dc98ee1b3ea689fee4, State: Running, Role: LEADER
I20260927 13:59:02.922721 4002 consensus_queue.cc:237] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00438db8e1f42dc98ee1b3ea689fee4" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42229 } }
I20260927 13:59:02.923513 3997 sys_catalog.cc:564] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:59:02.925664 4003 sys_catalog.cc:455] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f00438db8e1f42dc98ee1b3ea689fee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00438db8e1f42dc98ee1b3ea689fee4" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42229 } } }
I20260927 13:59:02.925710 4004 sys_catalog.cc:455] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f00438db8e1f42dc98ee1b3ea689fee4. Latest consensus state: current_term: 1 leader_uuid: "f00438db8e1f42dc98ee1b3ea689fee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00438db8e1f42dc98ee1b3ea689fee4" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42229 } } }
I20260927 13:59:02.926028 4003 sys_catalog.cc:458] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [sys.catalog]: This master's current role is: LEADER
I20260927 13:59:02.926098 4004 sys_catalog.cc:458] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4 [sys.catalog]: This master's current role is: LEADER
I20260927 13:59:02.931648 4007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:59:02.935773 4007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:59:02.944106 4007 catalog_manager.cc:1347] Generated new cluster ID: c7e5916bb03d4d99808d247f0a66d257
I20260927 13:59:02.944226 4007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:59:02.962040 4007 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:59:02.963944 4007 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:59:02.975634 4007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f00438db8e1f42dc98ee1b3ea689fee4: Generated new TSK 0
I20260927 13:59:02.976729 4007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:59:03.029639 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 3930
I20260927 13:59:03.030966 3992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:03.126860 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 3930
2026-09-27T13:59:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2023 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:59:03 dist-test-slave-gkks krb5kdc[4025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:59:03 dist-test-slave-gkks krb5kdc[4025](info): set up 2 sockets
Sep 27 13:59:03 dist-test-slave-gkks krb5kdc[4025](info): commencing operation
krb5kdc: starting...
W20260927 13:59:04.231839 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:59:04 dist-test-slave-gkks krb5kdc[4025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:59:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:59:04.478588 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39275
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:39275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:04.815866 4038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:04.816219 4038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:04.816396 4038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:04.823661 4038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:59:04.823823 4038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:04.823937 4038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:04.824043 4038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:59:04.824132 4038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:59:04.834065 4038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:39275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:04.835484 4038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:04.836851 4038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:04.849041 4044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:04.849115 4043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:04.849115 4046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:04.849749 4038 server_base.cc:1034] running on GCE node
I20260927 13:59:04.850626 4038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:04.852205 4038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:04.853495 4038 hybrid_clock.cc:648] HybridClock initialized: now 1790517544853445 us; error 59 us; skew 500 ppm
I20260927 13:59:04.853910 4038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:04 dist-test-slave-gkks krb5kdc[4025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517544, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:04.857141 4038 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:59:04.858695 4038 webserver.cc:460] Webserver started at http://127.22.26.254:45423/ using document root <none> and password file <none>
I20260927 13:59:04.859509 4038 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:04.859617 4038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:04.859876 4038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:59:04.862833 4038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "ad383ba76dfe42f39684e684dc2bf160"
format_stamp: "Formatted at 2026-09-27 13:59:04 on dist-test-slave-gkks"
server_key: "2b2f923b170661793b0ea5944a4ea579"
server_key_iv: "e3b5b7b42c08f77334f1cf6ff8497c61"
server_key_version: "encryptionkey@0"
I20260927 13:59:04.863698 4038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "ad383ba76dfe42f39684e684dc2bf160"
format_stamp: "Formatted at 2026-09-27 13:59:04 on dist-test-slave-gkks"
server_key: "2b2f923b170661793b0ea5944a4ea579"
server_key_iv: "e3b5b7b42c08f77334f1cf6ff8497c61"
server_key_version: "encryptionkey@0"
I20260927 13:59:04.868966 4038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:59:04.872541 4053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:04.874002 4038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:59:04.874183 4038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "ad383ba76dfe42f39684e684dc2bf160"
format_stamp: "Formatted at 2026-09-27 13:59:04 on dist-test-slave-gkks"
server_key: "2b2f923b170661793b0ea5944a4ea579"
server_key_iv: "e3b5b7b42c08f77334f1cf6ff8497c61"
server_key_version: "encryptionkey@0"
I20260927 13:59:04.874397 4038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:04.890539 4038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:04.894573 4038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:04.894871 4038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:04.913578 4038 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:39275
I20260927 13:59:04.913607 4104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:39275 every 8 connection(s)
I20260927 13:59:04.915031 4038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:59:04.918823 4105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:04.921360 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 4038
I20260927 13:59:04.921587 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:59:04.922423 22635 external_mini_cluster.cc:1428] Setting key 0105b8113d2c4b5311248fbe60648f53
I20260927 13:59:04.931926 4105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Bootstrap starting.
I20260927 13:59:04.935912 4105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:04.937309 4105 log.cc:824] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Log is configured to *not* fsync() on all Append() calls
Sep 27 13:59:04 dist-test-slave-gkks krb5kdc[4025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:04.940505 4105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: No bootstrap required, opened a new log
I20260927 13:59:04.946069 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:04.924646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.22.26.254:39275)
0927 13:59:04.925432 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:04.925444 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:04.925484 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:59:04.927160 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:04.927183 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:04.927572 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:04.927908 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:04.929780 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:04.931150 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:04.932920 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:04.933645 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:04.933777 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:04.938763 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:04.938771 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:04.938786 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:04.938821 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:04.941930 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:04.942666 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:04.942673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:04.942679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:04.942853 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:04.943309 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:04.943321 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:04.943324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:04.943861 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:04.944185 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:04.944662 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 13:59:04.945068 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":368,"threads_started":1}
I20260927 13:59:04.948558 4105 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } }
I20260927 13:59:04.948977 4105 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:04.949081 4105 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad383ba76dfe42f39684e684dc2bf160, State: Initialized, Role: FOLLOWER
I20260927 13:59:04.949558 4105 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } }
I20260927 13:59:04.949750 4105 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:59:04.949847 4105 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:59:04.949961 4105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:04.952701 4105 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } }
I20260927 13:59:04.953143 4105 leader_election.cc:302] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad383ba76dfe42f39684e684dc2bf160; no voters:
I20260927 13:59:04.953742 4105 leader_election.cc:288] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:59:04.953900 4110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:59:04.954668 4110 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 1 LEADER]: Becoming Leader. State: Replica: ad383ba76dfe42f39684e684dc2bf160, State: Running, Role: LEADER
I20260927 13:59:04.955268 4110 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } }
I20260927 13:59:04.955873 4105 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:59:04.958338 4111 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad383ba76dfe42f39684e684dc2bf160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } } }
I20260927 13:59:04.958362 4112 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad383ba76dfe42f39684e684dc2bf160. Latest consensus state: current_term: 1 leader_uuid: "ad383ba76dfe42f39684e684dc2bf160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } } }
I20260927 13:59:04.958729 4112 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: This master's current role is: LEADER
I20260927 13:59:04.958727 4111 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: This master's current role is: LEADER
I20260927 13:59:04.960278 4116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:59:04.965600 4116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:59:04.974357 4116 catalog_manager.cc:1347] Generated new cluster ID: 48f19a64b8df495fa62253deb3b1f699
I20260927 13:59:04.974474 4116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:59:04.985980 4116 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:59:04.987713 4116 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:59:04.998284 4116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Generated new TSK 0
I20260927 13:59:04.999320 4116 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:59:05.067739 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39275
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:05.432022 4131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:05.432283 4131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:05.432471 4131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:05.439666 4131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:05.439764 4131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:05.439980 4131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:59:05.452474 4131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:05.453749 4131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:05.455075 4131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:05.469280 4136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:05.469226 4139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:05.469321 4137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:05.470074 4131 server_base.cc:1034] running on GCE node
I20260927 13:59:05.470695 4131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:05.471616 4131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:05.472815 4131 hybrid_clock.cc:648] HybridClock initialized: now 1790517545472765 us; error 40 us; skew 500 ppm
I20260927 13:59:05.473129 4131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:05 dist-test-slave-gkks krb5kdc[4025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517545, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:05.476497 4131 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:59:05.477931 4131 webserver.cc:460] Webserver started at http://127.22.26.193:38521/ using document root <none> and password file <none>
I20260927 13:59:05.478641 4131 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:05.478741 4131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:05.478982 4131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:59:05.481834 4131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "6ee9f5e541a24f37bf449bc1cd141a28"
format_stamp: "Formatted at 2026-09-27 13:59:05 on dist-test-slave-gkks"
server_key: "e92a4233c63e8cba329614408d10e2b6"
server_key_iv: "c30fdff0cfedaf2b8f2ffffde0acca57"
server_key_version: "encryptionkey@0"
I20260927 13:59:05.482554 4131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "6ee9f5e541a24f37bf449bc1cd141a28"
format_stamp: "Formatted at 2026-09-27 13:59:05 on dist-test-slave-gkks"
server_key: "e92a4233c63e8cba329614408d10e2b6"
server_key_iv: "c30fdff0cfedaf2b8f2ffffde0acca57"
server_key_version: "encryptionkey@0"
I20260927 13:59:05.487494 4131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260927 13:59:05.490993 4146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:05.492398 4131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:59:05.492594 4131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "6ee9f5e541a24f37bf449bc1cd141a28"
format_stamp: "Formatted at 2026-09-27 13:59:05 on dist-test-slave-gkks"
server_key: "e92a4233c63e8cba329614408d10e2b6"
server_key_iv: "c30fdff0cfedaf2b8f2ffffde0acca57"
server_key_version: "encryptionkey@0"
I20260927 13:59:05.492803 4131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:05.535636 4131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:05.540045 4131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:05.540365 4131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:05.541361 4131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:05.542752 4131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:05.542825 4131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:05.542999 4131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:05.543064 4131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:05.575851 4131 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:35349
I20260927 13:59:05.575898 4258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:35349 every 8 connection(s)
I20260927 13:59:05.577111 4131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:59:05.584952 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 4131
I20260927 13:59:05.585184 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:59:05.585826 22635 external_mini_cluster.cc:1428] Setting key c3006819ec14a69018bc3e6aa73ac89c
Sep 27 13:59:05 dist-test-slave-gkks krb5kdc[4025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517545, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:05.599355 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:05.579762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:45955 (local address 127.22.26.254:39275)
0927 13:59:05.580231 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:05.580236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:05.581203 (+ 967us) server_negotiation.cc:407] Connection header received
0927 13:59:05.583013 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:05.583019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:05.583158 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:05.583278 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:05.584937 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:05.585886 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:05.587910 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:05.588524 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:05.588673 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:05.592220 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:05.592229 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:05.592237 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:05.592282 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:05.595882 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:05.596850 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:05.596854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:05.596857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:05.596975 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:05.597551 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:05.597563 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:05.597566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:05.597838 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:05.597979 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:05.598838 (+ 859us) server_negotiation.cc:299] Negotiation successful
0927 13:59:05.599018 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":244,"threads_started":1}
I20260927 13:59:05.600819 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:05.580372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:45955)
0927 13:59:05.581029 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:05.581078 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:05.582719 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:05.583509 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:05.583518 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:05.584009 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:59:05.584707 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:05.584737 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:05.586097 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:05.586102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:05.587740 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:05.587759 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:05.588768 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:05.588773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:05.588984 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:05.589743 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:59:05.589766 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:59:05.592003 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:59:05.596014 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:05.596025 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:05.596045 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:05.596726 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:05.597089 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:05.597097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:05.597104 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:05.597347 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:05.597983 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:59:05.597993 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:59:05.598448 (+ 455us) client_negotiation.cc:769] Sending connection context
0927 13:59:05.598746 (+ 298us) client_negotiation.cc:240] Negotiation successful
0927 13:59:05.599343 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":253,"threads_started":1}
I20260927 13:59:05.602367 4259 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39275
I20260927 13:59:05.602674 4259 heartbeater.cc:464] Registering TS with master...
I20260927 13:59:05.603464 4259 heartbeater.cc:511] Master 127.22.26.254:39275 requested a full tablet report, sending...
I20260927 13:59:05.605521 4070 ts_manager.cc:194] Registered new tserver with Master: 6ee9f5e541a24f37bf449bc1cd141a28 (127.22.26.193:35349)
I20260927 13:59:05.607072 4070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:45955
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:59:05.644214 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39275
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:05.970579 4265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:05.970899 4265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:05.971069 4265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:05.978444 4265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:05.978583 4265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:05.978768 4265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:59:05.988607 4265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:05.989910 4265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:05.991304 4265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:06.005653 4271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:06.005834 4270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:06.005681 4273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:06.006330 4265 server_base.cc:1034] running on GCE node
I20260927 13:59:06.007009 4265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:06.007974 4265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:06.009178 4265 hybrid_clock.cc:648] HybridClock initialized: now 1790517546009115 us; error 52 us; skew 500 ppm
I20260927 13:59:06.009524 4265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:06 dist-test-slave-gkks krb5kdc[4025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:06.012818 4265 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:59:06.014338 4265 webserver.cc:460] Webserver started at http://127.22.26.194:46149/ using document root <none> and password file <none>
I20260927 13:59:06.014992 4265 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:06.015156 4265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:06.015424 4265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:59:06.018319 4265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "3560ee761ed14470aefdb99eecf409eb"
format_stamp: "Formatted at 2026-09-27 13:59:06 on dist-test-slave-gkks"
server_key: "159d44b6607bd2b85757eea0701994c3"
server_key_iv: "65db2cd4498faa8594069dda63a97ae9"
server_key_version: "encryptionkey@0"
I20260927 13:59:06.019060 4265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "3560ee761ed14470aefdb99eecf409eb"
format_stamp: "Formatted at 2026-09-27 13:59:06 on dist-test-slave-gkks"
server_key: "159d44b6607bd2b85757eea0701994c3"
server_key_iv: "65db2cd4498faa8594069dda63a97ae9"
server_key_version: "encryptionkey@0"
I20260927 13:59:06.024117 4265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 13:59:06.027552 4280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:06.028959 4265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 13:59:06.029150 4265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "3560ee761ed14470aefdb99eecf409eb"
format_stamp: "Formatted at 2026-09-27 13:59:06 on dist-test-slave-gkks"
server_key: "159d44b6607bd2b85757eea0701994c3"
server_key_iv: "65db2cd4498faa8594069dda63a97ae9"
server_key_version: "encryptionkey@0"
I20260927 13:59:06.029346 4265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:06.046164 4265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:06.050496 4265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:06.050793 4265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:06.051816 4265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:06.053169 4265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:06.053268 4265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:06.053429 4265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:06.053511 4265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:06.086709 4265 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:33351
I20260927 13:59:06.086751 4392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:33351 every 8 connection(s)
I20260927 13:59:06.088155 4265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:59:06.098129 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 4265
I20260927 13:59:06.098327 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:59:06.098824 22635 external_mini_cluster.cc:1428] Setting key 3fb76e9c4a51f8927d7dc48a5a33bee9
Sep 27 13:59:06 dist-test-slave-gkks krb5kdc[4025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:06.108150 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.090677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:43939 (local address 127.22.26.254:39275)
0927 13:59:06.090851 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:06.090856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:06.092040 (+ 1184us) server_negotiation.cc:407] Connection header received
0927 13:59:06.093922 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:06.093928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:06.094022 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:06.094132 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:06.095667 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.096485 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.098075 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.098618 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.098748 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.102238 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:06.102249 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:06.102253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:06.102285 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:06.104995 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:06.105736 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:06.105742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:06.105750 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:06.105887 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:06.106331 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:06.106335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:06.106337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:06.106604 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:06.106761 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:06.107557 (+ 796us) server_negotiation.cc:299] Negotiation successful
0927 13:59:06.107791 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:59:06.109169 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.091165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:43939)
0927 13:59:06.091874 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:06.091922 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:06.093643 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:06.094369 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:06.094377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:06.094879 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:59:06.095500 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.095518 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.096640 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.096644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.097922 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.097933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.098853 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.098860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.099184 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.099872 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:59:06.099901 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:59:06.102010 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:59:06.105144 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:06.105153 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:06.105170 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:06.105597 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:06.105995 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:06.106000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:06.106004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:06.106203 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:06.106731 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:59:06.106739 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:59:06.107158 (+ 419us) client_negotiation.cc:769] Sending connection context
0927 13:59:06.107459 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 13:59:06.108010 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":283,"threads_started":1}
I20260927 13:59:06.110726 4393 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39275
I20260927 13:59:06.111066 4393 heartbeater.cc:464] Registering TS with master...
I20260927 13:59:06.111879 4393 heartbeater.cc:511] Master 127.22.26.254:39275 requested a full tablet report, sending...
I20260927 13:59:06.113517 4070 ts_manager.cc:194] Registered new tserver with Master: 3560ee761ed14470aefdb99eecf409eb (127.22.26.194:33351)
I20260927 13:59:06.114845 4070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:43939
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:59:06.152433 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39275
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:06.482784 4398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:06.483124 4398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:06.483253 4398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:06.490530 4398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:06.490635 4398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:06.490746 4398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:59:06.500862 4398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:06.502089 4398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:06.503453 4398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:06.516134 4403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:06.516340 4404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:06.516747 4406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:06.517549 4398 server_base.cc:1034] running on GCE node
I20260927 13:59:06.518173 4398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:06.519197 4398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:06.520430 4398 hybrid_clock.cc:648] HybridClock initialized: now 1790517546520378 us; error 43 us; skew 500 ppm
I20260927 13:59:06.520783 4398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:06 dist-test-slave-gkks krb5kdc[4025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:06.524011 4398 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:59:06.525421 4398 webserver.cc:460] Webserver started at http://127.22.26.195:34991/ using document root <none> and password file <none>
I20260927 13:59:06.526130 4398 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:06.526276 4398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:06.526576 4398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:59:06.529536 4398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "ffd528ed51f646f7b3e4ace6220225f5"
format_stamp: "Formatted at 2026-09-27 13:59:06 on dist-test-slave-gkks"
server_key: "ad509d673eb06481ed044b83648f45a1"
server_key_iv: "44e96af4983826853556ed349bd3fba0"
server_key_version: "encryptionkey@0"
I20260927 13:59:06.530318 4398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "ffd528ed51f646f7b3e4ace6220225f5"
format_stamp: "Formatted at 2026-09-27 13:59:06 on dist-test-slave-gkks"
server_key: "ad509d673eb06481ed044b83648f45a1"
server_key_iv: "44e96af4983826853556ed349bd3fba0"
server_key_version: "encryptionkey@0"
I20260927 13:59:06.535526 4398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 13:59:06.538899 4413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:06.540469 4398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:59:06.540658 4398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "ffd528ed51f646f7b3e4ace6220225f5"
format_stamp: "Formatted at 2026-09-27 13:59:06 on dist-test-slave-gkks"
server_key: "ad509d673eb06481ed044b83648f45a1"
server_key_iv: "44e96af4983826853556ed349bd3fba0"
server_key_version: "encryptionkey@0"
I20260927 13:59:06.540863 4398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:06.557951 4398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:06.562439 4398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:06.562742 4398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:06.563836 4398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:06.565276 4398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:06.565378 4398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:06.565536 4398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:06.565609 4398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:06.600519 4398 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:43269
I20260927 13:59:06.600545 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:43269 every 8 connection(s)
I20260927 13:59:06.601909 4398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:59:06.604894 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 4398
I20260927 13:59:06.605088 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:59:06.605651 22635 external_mini_cluster.cc:1428] Setting key 877ab74d149a4eabc72e61a94ea56f8b
I20260927 13:59:06.610008 4259 heartbeater.cc:503] Master 127.22.26.254:39275 was elected leader, sending a full tablet report...
Sep 27 13:59:06 dist-test-slave-gkks krb5kdc[4025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:06.623797 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.604681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:46873 (local address 127.22.26.254:39275)
0927 13:59:06.604857 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:06.604861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:06.606521 (+ 1660us) server_negotiation.cc:407] Connection header received
0927 13:59:06.608682 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:06.608687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:06.608779 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:06.608896 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:06.610682 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.611839 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.613436 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.613874 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.614006 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.617627 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:06.617632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:06.617635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:06.617668 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:06.620636 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:06.621476 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:06.621483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:06.621486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:06.621660 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:06.622087 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:06.622091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:06.622093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:06.622388 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:06.622524 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:06.623343 (+ 819us) server_negotiation.cc:299] Negotiation successful
0927 13:59:06.623540 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:59:06.624791 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.605388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:46873)
0927 13:59:06.606245 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:06.606355 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:06.608418 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:06.609146 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:06.609154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:06.609794 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:59:06.610484 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.610510 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.611984 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.611988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.613253 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.613265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.614162 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.614167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.614377 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.615197 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:59:06.615220 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:59:06.617401 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:59:06.620786 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:06.620793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:06.620806 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:06.621293 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:06.621782 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:06.621786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:06.621788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:06.621975 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:06.622542 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:59:06.622548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:59:06.622895 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 13:59:06.623221 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 13:59:06.623657 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":340,"threads_started":1}
I20260927 13:59:06.626183 4526 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39275
I20260927 13:59:06.626469 4526 heartbeater.cc:464] Registering TS with master...
I20260927 13:59:06.627220 4526 heartbeater.cc:511] Master 127.22.26.254:39275 requested a full tablet report, sending...
I20260927 13:59:06.628554 4070 ts_manager.cc:194] Registered new tserver with Master: ffd528ed51f646f7b3e4ace6220225f5 (127.22.26.195:43269)
I20260927 13:59:06.629355 4070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:46873
I20260927 13:59:06.633677 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:59:06.651590 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.638319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.22.26.254:39275)
0927 13:59:06.638499 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:06.638506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:06.638991 (+ 485us) server_negotiation.cc:407] Connection header received
0927 13:59:06.639230 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:06.639235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:06.639304 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:06.639407 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:06.640845 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.641560 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.642913 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.643391 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.643526 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.646103 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:06.646107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:06.646111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:06.646140 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:06.648924 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:06.649595 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:06.649601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:06.649606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:06.649734 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:06.650118 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:06.650123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:06.650127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:06.650430 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:06.650738 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:06.651015 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 13:59:06.651258 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:59:06.655627 4070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:59:06.662499 4070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:59:06.688490 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.674732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35349 (local address 127.0.0.1:57358)
0927 13:59:06.676145 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:06.676199 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:06.676343 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:06.677243 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:06.677250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:06.677281 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:06.677736 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.677754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.680679 (+ 2925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.680688 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.683949 (+ 3261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.683970 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.686907 (+ 2937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.686917 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.687255 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.687330 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 13:59:06.687504 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 13:59:06.687792 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"mutex_wait_us":1,"thread_start_us":958,"threads_started":1}
I20260927 13:59:06.688517 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.676652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:33351 (local address 127.0.0.1:37308)
0927 13:59:06.677201 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:06.677226 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:06.677382 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:06.678850 (+ 1468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:06.678858 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:06.678895 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:06.679484 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.679500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.682223 (+ 2723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.682232 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.685144 (+ 2912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.685164 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.687653 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.687660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.687857 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.687890 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:59:06.687952 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 13:59:06.688076 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":254,"threads_started":1}
I20260927 13:59:06.689430 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.675007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.22.26.193:35349)
0927 13:59:06.675783 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:06.675794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:06.676297 (+ 503us) server_negotiation.cc:407] Connection header received
0927 13:59:06.676553 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:06.676561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:06.676844 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:06.677069 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:06.678183 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.680457 (+ 2274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.684193 (+ 3736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.686236 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.686411 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.686749 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:06.688838 (+ 2089us) server_negotiation.cc:299] Negotiation successful
0927 13:59:06.689055 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":350,"threads_started":1}
I20260927 13:59:06.689865 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.676907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.22.26.194:33351)
0927 13:59:06.678038 (+ 1131us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:06.678049 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:06.678085 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:59:06.678212 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:06.678224 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:06.678526 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:06.679755 (+ 1229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:06.680020 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.682025 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.685424 (+ 3399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.687393 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.688853 (+ 1460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.689086 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:06.689166 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 13:59:06.689348 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":978,"thread_start_us":849,"threads_started":1}
I20260927 13:59:06.693486 4194 tablet_service.cc:1467] Processing CreateTablet for tablet c784bc1f07b1443480cc885617b7af90 (DEFAULT_TABLE table=test-table [id=cc028e33795344cb9caaa2e92e64ff6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:59:06.693495 4328 tablet_service.cc:1467] Processing CreateTablet for tablet c784bc1f07b1443480cc885617b7af90 (DEFAULT_TABLE table=test-table [id=cc028e33795344cb9caaa2e92e64ff6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:59:06.694962 4194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c784bc1f07b1443480cc885617b7af90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:06.694962 4328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c784bc1f07b1443480cc885617b7af90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:06.697853 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.677253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:43269 (local address 127.0.0.1:51238)
0927 13:59:06.687401 (+ 10148us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:06.687421 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:06.687540 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:06.690305 (+ 2765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:06.690309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:06.690338 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:06.690683 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.690696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.692711 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.692716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.694717 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.694733 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.697026 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.697032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.697208 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.697242 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 13:59:06.697346 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 13:59:06.697494 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10003,"thread_start_us":10380,"threads_started":1}
I20260927 13:59:06.697909 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.677853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.22.26.195:43269)
0927 13:59:06.689662 (+ 11809us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:06.689671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:06.689707 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:59:06.689802 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:06.689811 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:06.690095 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:06.690297 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:06.690906 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.692550 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.694927 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.696759 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.696906 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.697170 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:06.697407 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 13:59:06.697571 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11282,"thread_start_us":11109,"threads_started":1}
I20260927 13:59:06.702198 4461 tablet_service.cc:1467] Processing CreateTablet for tablet c784bc1f07b1443480cc885617b7af90 (DEFAULT_TABLE table=test-table [id=cc028e33795344cb9caaa2e92e64ff6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:59:06.703740 4461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c784bc1f07b1443480cc885617b7af90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:06.707433 4544 tablet_bootstrap.cc:492] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb: Bootstrap starting.
I20260927 13:59:06.709537 4545 tablet_bootstrap.cc:492] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28: Bootstrap starting.
I20260927 13:59:06.711663 4544 tablet_bootstrap.cc:654] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:06.712991 4544 log.cc:824] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:06.713506 4545 tablet_bootstrap.cc:654] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:06.714696 4545 log.cc:824] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:06.716099 4544 tablet_bootstrap.cc:492] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb: No bootstrap required, opened a new log
I20260927 13:59:06.716382 4544 ts_tablet_manager.cc:1397] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 13:59:06.717722 4545 tablet_bootstrap.cc:492] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28: No bootstrap required, opened a new log
I20260927 13:59:06.717734 4548 tablet_bootstrap.cc:492] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5: Bootstrap starting.
I20260927 13:59:06.718079 4545 ts_tablet_manager.cc:1397] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260927 13:59:06.720728 4548 tablet_bootstrap.cc:654] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:06.721674 4548 log.cc:824] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:06.724148 4548 tablet_bootstrap.cc:492] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5: No bootstrap required, opened a new log
I20260927 13:59:06.724412 4548 ts_tablet_manager.cc:1397] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 13:59:06.728374 4544 raft_consensus.cc:348] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:06.728852 4544 raft_consensus.cc:374] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:06.728667 4545 raft_consensus.cc:348] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:06.729012 4544 raft_consensus.cc:729] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3560ee761ed14470aefdb99eecf409eb, State: Initialized, Role: FOLLOWER
I20260927 13:59:06.729076 4545 raft_consensus.cc:374] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:06.729206 4545 raft_consensus.cc:729] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee9f5e541a24f37bf449bc1cd141a28, State: Initialized, Role: FOLLOWER
I20260927 13:59:06.729660 4544 consensus_queue.cc:260] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:06.729823 4545 consensus_queue.cc:260] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:06.730876 4393 heartbeater.cc:503] Master 127.22.26.254:39275 was elected leader, sending a full tablet report...
I20260927 13:59:06.731889 4544 ts_tablet_manager.cc:1428] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 13:59:06.733717 4545 ts_tablet_manager.cc:1428] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 13:59:06.734596 4548 raft_consensus.cc:348] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:06.735144 4548 raft_consensus.cc:374] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:06.735265 4548 raft_consensus.cc:729] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd528ed51f646f7b3e4ace6220225f5, State: Initialized, Role: FOLLOWER
I20260927 13:59:06.735996 4548 consensus_queue.cc:260] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:06.737170 4526 heartbeater.cc:503] Master 127.22.26.254:39275 was elected leader, sending a full tablet report...
I20260927 13:59:06.737767 4548 ts_tablet_manager.cc:1428] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260927 13:59:06.778115 4551 raft_consensus.cc:482] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:59:06.778438 4551 raft_consensus.cc:504] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:06.780345 4551 leader_election.cc:288] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3560ee761ed14470aefdb99eecf409eb (127.22.26.194:33351), ffd528ed51f646f7b3e4ace6220225f5 (127.22.26.195:43269)
I20260927 13:59:06.788754 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.780820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:48391 (local address 127.22.26.194:33351)
0927 13:59:06.781005 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:06.781010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:06.781575 (+ 565us) server_negotiation.cc:407] Connection header received
0927 13:59:06.781858 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:06.781863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:06.781943 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:06.782053 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:06.782809 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.783911 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.786579 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.787719 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.787861 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.787912 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:06.788337 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 13:59:06.788484 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:59:06.788827 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.781950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:43269 (local address 127.22.26.195:43269)
0927 13:59:06.782157 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:06.782163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:06.782386 (+ 223us) server_negotiation.cc:407] Connection header received
0927 13:59:06.782624 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:06.782627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:06.782684 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:06.782778 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:06.783267 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.784242 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.786628 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.787717 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.787895 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.787966 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:06.788453 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 13:59:06.788597 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 13:59:06.788846 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.780785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:33351 (local address 127.22.26.193:48391)
0927 13:59:06.781466 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:06.781488 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:06.781632 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:06.782220 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:06.782226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:06.782252 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:06.782680 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.782691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.784056 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.784060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.786382 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.786410 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.787922 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.787929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.788169 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.788195 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 13:59:06.788272 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 13:59:06.788381 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":54,"thread_start_us":361,"threads_started":1}
I20260927 13:59:06.788950 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.781636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:43269 (local address 127.22.26.193:43269)
0927 13:59:06.782284 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:06.782304 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:06.782496 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:06.782816 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:06.782820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:06.782849 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:06.783127 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.783135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.784369 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.784373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.786434 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:06.786447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.788078 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.788083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:06.788238 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.788300 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 13:59:06.788396 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 13:59:06.788513 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":1,"spinlock_wait_cycles":24448,"thread_start_us":353,"threads_started":1}
I20260927 13:59:06.790055 4481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c784bc1f07b1443480cc885617b7af90" candidate_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd528ed51f646f7b3e4ace6220225f5" is_pre_election: true
I20260927 13:59:06.790153 4348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c784bc1f07b1443480cc885617b7af90" candidate_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3560ee761ed14470aefdb99eecf409eb" is_pre_election: true
I20260927 13:59:06.790665 4348 raft_consensus.cc:2454] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ee9f5e541a24f37bf449bc1cd141a28 in term 0.
I20260927 13:59:06.790664 4481 raft_consensus.cc:2454] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ee9f5e541a24f37bf449bc1cd141a28 in term 0.
I20260927 13:59:06.791397 4149 leader_election.cc:302] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ee9f5e541a24f37bf449bc1cd141a28, ffd528ed51f646f7b3e4ace6220225f5; no voters:
I20260927 13:59:06.791852 4551 raft_consensus.cc:2781] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:59:06.792038 4551 raft_consensus.cc:482] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:59:06.792152 4551 raft_consensus.cc:3037] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:06.794384 4551 raft_consensus.cc:504] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:06.794972 4551 leader_election.cc:288] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [CANDIDATE]: Term 1 election: Requested vote from peers 3560ee761ed14470aefdb99eecf409eb (127.22.26.194:33351), ffd528ed51f646f7b3e4ace6220225f5 (127.22.26.195:43269)
I20260927 13:59:06.795526 4348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c784bc1f07b1443480cc885617b7af90" candidate_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3560ee761ed14470aefdb99eecf409eb"
I20260927 13:59:06.795608 4481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c784bc1f07b1443480cc885617b7af90" candidate_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd528ed51f646f7b3e4ace6220225f5"
I20260927 13:59:06.795727 4348 raft_consensus.cc:3037] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:06.795913 4481 raft_consensus.cc:3037] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:06.798211 4348 raft_consensus.cc:2454] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ee9f5e541a24f37bf449bc1cd141a28 in term 1.
I20260927 13:59:06.798229 4481 raft_consensus.cc:2454] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ee9f5e541a24f37bf449bc1cd141a28 in term 1.
I20260927 13:59:06.798653 4149 leader_election.cc:302] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3560ee761ed14470aefdb99eecf409eb, 6ee9f5e541a24f37bf449bc1cd141a28; no voters:
I20260927 13:59:06.798959 4551 raft_consensus.cc:2781] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:59:06.799582 4551 raft_consensus.cc:686] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 1 LEADER]: Becoming Leader. State: Replica: 6ee9f5e541a24f37bf449bc1cd141a28, State: Running, Role: LEADER
I20260927 13:59:06.800055 4551 consensus_queue.cc:237] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:06.804797 4070 catalog_manager.cc:5495] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ee9f5e541a24f37bf449bc1cd141a28 (127.22.26.193). New cstate: current_term: 1 leader_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } health_report { overall_health: UNKNOWN } } }
I20260927 13:59:06.823935 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.818280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.22.26.193:35349)
0927 13:59:06.818487 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:06.818493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:06.818613 (+ 120us) server_negotiation.cc:407] Connection header received
0927 13:59:06.818862 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:06.818866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:06.818953 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:06.819140 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:59:06.819512 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.820221 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.821634 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.822044 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.822180 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.822694 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:59:06.823357 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:59:06.823517 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:06.823581 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 13:59:06.823662 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 13:59:06.829558 4260 tablet.cc:2369] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:59:06.832428 4348 raft_consensus.cc:1261] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb [term 1 FOLLOWER]: Refusing update from remote peer 6ee9f5e541a24f37bf449bc1cd141a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:59:06.832468 4481 raft_consensus.cc:1261] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5 [term 1 FOLLOWER]: Refusing update from remote peer 6ee9f5e541a24f37bf449bc1cd141a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:59:06.833472 4555 consensus_queue.cc:1035] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:59:06.833981 4551 consensus_queue.cc:1035] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 13:59:06.840130 4394 tablet.cc:2369] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:59:06.847299 4558 mvcc.cc:205] Tried to move back new op lower bound from 7333959871815200768 to 7333959871696490496. Current Snapshot: MvccSnapshot[applied={T|T < 7333959871815200768}]
I20260927 13:59:06.872720 4069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:59:06.873087 4069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:59:06.877307 4069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Sending DeleteTablet for 3 replicas of tablet c784bc1f07b1443480cc885617b7af90
I20260927 13:59:06.878466 4194 tablet_service.cc:1514] Processing DeleteTablet for tablet c784bc1f07b1443480cc885617b7af90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:59:06 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:57358
I20260927 13:59:06.878849 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 4131
I20260927 13:59:06.879379 4461 tablet_service.cc:1514] Processing DeleteTablet for tablet c784bc1f07b1443480cc885617b7af90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:59:06 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:51238
I20260927 13:59:06.880048 4568 tablet_replica.cc:330] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28: stopping tablet replica
I20260927 13:59:06.879047 4328 tablet_service.cc:1514] Processing DeleteTablet for tablet c784bc1f07b1443480cc885617b7af90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:59:06 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:37308
I20260927 13:59:06.880491 4568 raft_consensus.cc:2229] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:59:06.881219 4568 raft_consensus.cc:2258] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:59:06.884526 4568 ts_tablet_manager.cc:1905] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:59:06.890880 4571 tablet_replica.cc:330] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb: stopping tablet replica
I20260927 13:59:06.891479 4571 raft_consensus.cc:2229] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:59:06.892086 4571 raft_consensus.cc:2258] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:59:06.893590 4568 ts_tablet_manager.cc:1918] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:59:06.893805 4568 log.cc:1196] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/c784bc1f07b1443480cc885617b7af90
I20260927 13:59:06.894474 4568 ts_tablet_manager.cc:1939] T c784bc1f07b1443480cc885617b7af90 P 6ee9f5e541a24f37bf449bc1cd141a28: Deleting consensus metadata
I20260927 13:59:06.894497 4571 ts_tablet_manager.cc:1905] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:59:06.895452 4569 tablet_replica.cc:330] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5: stopping tablet replica
I20260927 13:59:06.895934 4569 raft_consensus.cc:2229] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:59:06.896494 4056 catalog_manager.cc:4841] TS 6ee9f5e541a24f37bf449bc1cd141a28 (127.22.26.193:35349): tablet c784bc1f07b1443480cc885617b7af90 (table test-table [id=cc028e33795344cb9caaa2e92e64ff6c]) successfully deleted
I20260927 13:59:06.896566 4569 raft_consensus.cc:2258] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:59:06.898586 4569 ts_tablet_manager.cc:1905] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:59:06.903229 4571 ts_tablet_manager.cc:1918] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:59:06.903456 4571 log.cc:1196] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/c784bc1f07b1443480cc885617b7af90
I20260927 13:59:06.904065 4571 ts_tablet_manager.cc:1939] T c784bc1f07b1443480cc885617b7af90 P 3560ee761ed14470aefdb99eecf409eb: Deleting consensus metadata
I20260927 13:59:06.905596 4569 ts_tablet_manager.cc:1918] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:59:06.905802 4569 log.cc:1196] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/c784bc1f07b1443480cc885617b7af90
I20260927 13:59:06.906400 4569 ts_tablet_manager.cc:1939] T c784bc1f07b1443480cc885617b7af90 P ffd528ed51f646f7b3e4ace6220225f5: Deleting consensus metadata
I20260927 13:59:06.906370 4056 catalog_manager.cc:4841] TS 3560ee761ed14470aefdb99eecf409eb (127.22.26.194:33351): tablet c784bc1f07b1443480cc885617b7af90 (table test-table [id=cc028e33795344cb9caaa2e92e64ff6c]) successfully deleted
Sep 27 13:59:06 dist-test-slave-gkks krb5kdc[4025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:59:06.908201 4056 catalog_manager.cc:4841] TS ffd528ed51f646f7b3e4ace6220225f5 (127.22.26.195:43269): tablet c784bc1f07b1443480cc885617b7af90 (table test-table [id=cc028e33795344cb9caaa2e92e64ff6c]) successfully deleted
I20260927 13:59:06.915268 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:06.880148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.22.26.193:35349)
0927 13:59:06.880315 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:06.880322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:06.882189 (+ 1867us) server_negotiation.cc:407] Connection header received
0927 13:59:06.894694 (+ 12505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:06.894704 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:06.894830 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:06.894924 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:06.899535 (+ 4611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.900721 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.902719 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:06.903497 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:06.903611 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:06.909033 (+ 5422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:06.909046 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:06.909060 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:06.909106 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:06.912224 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:06.912996 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:06.913004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:06.913006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:06.913152 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:06.913600 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:06.913612 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:06.913616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:06.913991 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:06.914290 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:06.914597 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 13:59:06.914837 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:59:06.915961 4254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:07.019445 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 4131
I20260927 13:59:07.038272 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 4265
Sep 27 13:59:07 dist-test-slave-gkks krb5kdc[4025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:59:07.058084 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:07.039067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.22.26.194:33351)
0927 13:59:07.039301 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:07.039307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:07.039336 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:59:07.039563 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:07.039568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:07.039640 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:07.039725 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:07.041057 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:07.041775 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:07.043193 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:07.043627 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:07.043723 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:07.052419 (+ 8696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:07.052434 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:07.052450 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:07.052489 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:07.055380 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:07.056020 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:07.056027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:07.056030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:07.056157 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:07.056582 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:07.056586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:07.056588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:07.057059 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:07.057275 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:07.057565 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 13:59:07.057723 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 13:59:07.058815 4388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:07.147773 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 4265
I20260927 13:59:07.162103 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 4398
Sep 27 13:59:07 dist-test-slave-gkks krb5kdc[4025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:59:07.177034 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:07.162880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51242 (local address 127.22.26.195:43269)
0927 13:59:07.163129 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:07.163135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:07.163164 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:59:07.163330 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:07.163334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:07.163400 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:07.163521 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:07.164886 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:07.165611 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:07.166796 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:07.167319 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:07.167422 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:07.171692 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:07.171708 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:07.171724 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:07.171758 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:07.174555 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:07.175271 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:07.175279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:07.175281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:07.175417 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:07.175781 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:07.175784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:07.175786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:07.176117 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:07.176334 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:07.176629 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 13:59:07.176793 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 13:59:07.177676 4521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:07.267961 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 4398
I20260927 13:59:07.284715 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 4038
I20260927 13:59:07.285951 4100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:07.398478 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 4038
I20260927 13:59:07.420032 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39275
--webserver_interface=127.22.26.254
--webserver_port=45423
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:39275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:07.752002 4576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:07.752345 4576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:07.752545 4576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:07.759681 4576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:59:07.759829 4576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:07.759939 4576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:07.760046 4576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:59:07.760118 4576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:59:07.770031 4576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:39275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=45423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:07.771582 4576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:07.773285 4576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:07.799036 4583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:07.799058 4585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:07.799144 4582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:07.800247 4576 server_base.cc:1034] running on GCE node
I20260927 13:59:07.801270 4576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:07.803149 4576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:07.804421 4576 hybrid_clock.cc:648] HybridClock initialized: now 1790517547804346 us; error 56 us; skew 500 ppm
I20260927 13:59:07.804884 4576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:07 dist-test-slave-gkks krb5kdc[4025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517547, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:07.808907 4576 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:59:07.810757 4576 webserver.cc:460] Webserver started at http://127.22.26.254:45423/ using document root <none> and password file <none>
I20260927 13:59:07.811669 4576 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:07.811859 4576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:07.818996 4576 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 13:59:07.822542 4592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:07.824255 4576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:59:07.824486 4576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "ad383ba76dfe42f39684e684dc2bf160"
format_stamp: "Formatted at 2026-09-27 13:59:04 on dist-test-slave-gkks"
server_key: "2b2f923b170661793b0ea5944a4ea579"
server_key_iv: "e3b5b7b42c08f77334f1cf6ff8497c61"
server_key_version: "encryptionkey@0"
I20260927 13:59:07.825687 4576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:07.846648 4576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:07.851068 4576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:07.851402 4576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:07.871929 4576 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:39275
I20260927 13:59:07.871951 4643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:39275 every 8 connection(s)
I20260927 13:59:07.873622 4576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:59:07.875003 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 4576
I20260927 13:59:07.875905 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:35349
--local_ip_for_outbound_sockets=127.22.26.193
--tserver_master_addrs=127.22.26.254:39275
--webserver_port=38521
--webserver_interface=127.22.26.193
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:59:07.881145 4644 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:59:07.884894 4644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Bootstrap starting.
I20260927 13:59:07.914788 4644 log.cc:824] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:07.930665 4644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:59:07.931392 4644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Bootstrap complete.
I20260927 13:59:07.946580 4644 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } }
I20260927 13:59:07.947921 4644 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad383ba76dfe42f39684e684dc2bf160, State: Initialized, Role: FOLLOWER
I20260927 13:59:07.948680 4644 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } }
I20260927 13:59:07.948951 4644 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:59:07.949103 4644 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:59:07.949295 4644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:59:07.953385 4644 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } }
I20260927 13:59:07.953915 4644 leader_election.cc:302] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad383ba76dfe42f39684e684dc2bf160; no voters:
I20260927 13:59:07.954672 4644 leader_election.cc:288] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 13:59:07.954815 4648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:59:07.956147 4648 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 2 LEADER]: Becoming Leader. State: Replica: ad383ba76dfe42f39684e684dc2bf160, State: Running, Role: LEADER
I20260927 13:59:07.956771 4648 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } }
I20260927 13:59:07.957274 4644 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:59:07.959239 4650 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad383ba76dfe42f39684e684dc2bf160. Latest consensus state: current_term: 2 leader_uuid: "ad383ba76dfe42f39684e684dc2bf160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } } }
I20260927 13:59:07.959712 4650 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: This master's current role is: LEADER
I20260927 13:59:07.959470 4649 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ad383ba76dfe42f39684e684dc2bf160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } } }
I20260927 13:59:07.960057 4649 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: This master's current role is: LEADER
I20260927 13:59:07.970125 4655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:59:07.979339 4655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:59:07.981082 4655 catalog_manager.cc:1259] Loaded cluster ID: 48f19a64b8df495fa62253deb3b1f699
I20260927 13:59:07.981274 4655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:59:07.985126 4655 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 13:59:08.014292 4655 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 13:59:08.036432 4655 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:59:08.038427 4655 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Loaded TSK: 0
I20260927 13:59:08.040174 4655 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 13:59:08.267717 4646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:08.267990 4646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:08.268159 4646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:08.276227 4646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:08.276332 4646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:08.276508 4646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:59:08.286276 4646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:35349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=38521
--tserver_master_addrs=127.22.26.254:39275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:08.287657 4646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:08.289021 4646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:08.302150 4672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:08.302237 4671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:08.302728 4674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:08.304001 4646 server_base.cc:1034] running on GCE node
I20260927 13:59:08.304769 4646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:08.305990 4646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:08.307197 4646 hybrid_clock.cc:648] HybridClock initialized: now 1790517548307128 us; error 61 us; skew 500 ppm
I20260927 13:59:08.307593 4646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:08 dist-test-slave-gkks krb5kdc[4025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:08.318184 4646 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:59:08.320225 4646 webserver.cc:460] Webserver started at http://127.22.26.193:38521/ using document root <none> and password file <none>
I20260927 13:59:08.321260 4646 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:08.321432 4646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:08.329286 4646 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:59:08.332878 4681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:08.334537 4646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:59:08.334748 4646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "6ee9f5e541a24f37bf449bc1cd141a28"
format_stamp: "Formatted at 2026-09-27 13:59:05 on dist-test-slave-gkks"
server_key: "e92a4233c63e8cba329614408d10e2b6"
server_key_iv: "c30fdff0cfedaf2b8f2ffffde0acca57"
server_key_version: "encryptionkey@0"
I20260927 13:59:08.336329 4646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:08.365254 4646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:08.371199 4646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:08.371524 4646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:08.372843 4646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:08.374657 4646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:08.374794 4646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:08.374936 4646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:08.375029 4646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:08.413815 4646 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:35349
I20260927 13:59:08.413827 4793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:35349 every 8 connection(s)
I20260927 13:59:08.415890 4646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:59:08.424804 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 4646
I20260927 13:59:08.425623 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:33351
--local_ip_for_outbound_sockets=127.22.26.194
--tserver_master_addrs=127.22.26.254:39275
--webserver_port=46149
--webserver_interface=127.22.26.194
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:59:08 dist-test-slave-gkks krb5kdc[4025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:08.442242 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:08.419141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:51495 (local address 127.22.26.254:39275)
0927 13:59:08.419798 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:08.419820 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:08.420635 (+ 815us) server_negotiation.cc:407] Connection header received
0927 13:59:08.422778 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:08.422797 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:08.423273 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:08.423659 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:08.425456 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:08.426808 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:08.429303 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:08.429961 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:08.430110 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:08.434364 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:08.434371 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:08.434384 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:08.434419 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:08.437644 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:08.438364 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:08.438371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:08.438377 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:08.438531 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:08.438978 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:08.438987 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:08.438990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:08.439520 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:08.439734 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:08.440514 (+ 780us) server_negotiation.cc:299] Negotiation successful
0927 13:59:08.440906 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":390,"threads_started":1}
I20260927 13:59:08.442243 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:08.419408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:51495)
0927 13:59:08.420415 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:08.420477 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:08.422208 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:08.423776 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:08.423785 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:08.424423 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:59:08.425224 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:08.425253 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:08.426978 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:08.426985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:08.429132 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:08.429149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:08.430200 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:08.430210 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:08.430570 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:08.431414 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:59:08.431453 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:59:08.434097 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:59:08.437780 (+ 3683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:08.437789 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:08.437802 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:08.438211 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:08.438649 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:08.438655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:08.438658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:08.438853 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:08.439668 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:59:08.439677 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:59:08.440013 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 13:59:08.440286 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 13:59:08.440852 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":512,"threads_started":1}
I20260927 13:59:08.445308 4794 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39275
I20260927 13:59:08.445667 4794 heartbeater.cc:464] Registering TS with master...
I20260927 13:59:08.446640 4794 heartbeater.cc:511] Master 127.22.26.254:39275 requested a full tablet report, sending...
I20260927 13:59:08.448961 4609 ts_manager.cc:194] Registered new tserver with Master: 6ee9f5e541a24f37bf449bc1cd141a28 (127.22.26.193:35349)
I20260927 13:59:08.450449 4609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:51495
W20260927 13:59:08.753362 4798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:08.753700 4798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:08.753882 4798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:08.761317 4798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:08.761466 4798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:08.761641 4798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:59:08.771749 4798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:33351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=46149
--tserver_master_addrs=127.22.26.254:39275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:08.773197 4798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:08.774576 4798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:08.788046 4803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:08.788000 4806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:08.788077 4804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:08.789203 4798 server_base.cc:1034] running on GCE node
I20260927 13:59:08.789889 4798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:08.790851 4798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:08.792053 4798 hybrid_clock.cc:648] HybridClock initialized: now 1790517548791997 us; error 47 us; skew 500 ppm
I20260927 13:59:08.792397 4798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:08 dist-test-slave-gkks krb5kdc[4025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:08.795992 4798 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:59:08.797775 4798 webserver.cc:460] Webserver started at http://127.22.26.194:46149/ using document root <none> and password file <none>
I20260927 13:59:08.798586 4798 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:08.798731 4798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:08.804242 4798 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:59:08.807039 4813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:08.808652 4798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:59:08.808833 4798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "3560ee761ed14470aefdb99eecf409eb"
format_stamp: "Formatted at 2026-09-27 13:59:06 on dist-test-slave-gkks"
server_key: "159d44b6607bd2b85757eea0701994c3"
server_key_iv: "65db2cd4498faa8594069dda63a97ae9"
server_key_version: "encryptionkey@0"
I20260927 13:59:08.809880 4798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:08.844337 4798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:08.848690 4798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:08.848960 4798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:08.850003 4798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:08.851414 4798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:08.851511 4798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:08.851604 4798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:08.851658 4798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:08.887266 4798 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:33351
I20260927 13:59:08.887311 4925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:33351 every 8 connection(s)
I20260927 13:59:08.888806 4798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:59:08.891121 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 4798
I20260927 13:59:08.891901 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:43269
--local_ip_for_outbound_sockets=127.22.26.195
--tserver_master_addrs=127.22.26.254:39275
--webserver_port=34991
--webserver_interface=127.22.26.195
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:59:08 dist-test-slave-gkks krb5kdc[4025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:08.914259 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:08.891905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:45807 (local address 127.22.26.254:39275)
0927 13:59:08.892114 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:08.892121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:08.893704 (+ 1583us) server_negotiation.cc:407] Connection header received
0927 13:59:08.895877 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:08.895884 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:08.896008 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:08.896133 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:08.897917 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:08.898843 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:08.901789 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:08.902221 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:08.902326 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:08.907302 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:08.907309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:08.907312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:08.907382 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:08.910316 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:08.911477 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:08.911482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:08.911485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:08.911627 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:08.912198 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:08.912201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:08.912203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:08.912481 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:08.912619 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:08.913728 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0927 13:59:08.913914 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:59:08.916213 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:08.892590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:45807)
0927 13:59:08.893523 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:08.893580 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:08.895575 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:08.896402 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:08.896411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:08.896966 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:59:08.897728 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:08.897749 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:08.899121 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:08.899129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:08.901591 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:08.901611 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:08.902467 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:08.902476 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:08.902838 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:08.903687 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:59:08.903724 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:59:08.907014 (+ 3290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:59:08.910492 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:08.910506 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:08.910529 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:08.911330 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:08.911757 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:08.911764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:08.911770 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:08.912075 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:08.912624 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:59:08.912637 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:59:08.913249 (+ 612us) client_negotiation.cc:769] Sending connection context
0927 13:59:08.913618 (+ 369us) client_negotiation.cc:240] Negotiation successful
0927 13:59:08.914372 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":361,"threads_started":1}
I20260927 13:59:08.917925 4926 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39275
I20260927 13:59:08.918246 4926 heartbeater.cc:464] Registering TS with master...
I20260927 13:59:08.919000 4926 heartbeater.cc:511] Master 127.22.26.254:39275 requested a full tablet report, sending...
I20260927 13:59:08.920636 4609 ts_manager.cc:194] Registered new tserver with Master: 3560ee761ed14470aefdb99eecf409eb (127.22.26.194:33351)
I20260927 13:59:08.921466 4609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:45807
W20260927 13:59:09.224731 4929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:09.225078 4929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:09.225256 4929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:09.232676 4929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:09.232820 4929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:09.233008 4929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:59:09.243016 4929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:43269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=34991
--tserver_master_addrs=127.22.26.254:39275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:09.244496 4929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:09.245834 4929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:09.259367 4934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:09.259420 4937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:09.259398 4935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:09.260108 4929 server_base.cc:1034] running on GCE node
I20260927 13:59:09.260697 4929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:09.261628 4929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:09.262838 4929 hybrid_clock.cc:648] HybridClock initialized: now 1790517549262784 us; error 44 us; skew 500 ppm
I20260927 13:59:09.263269 4929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:09 dist-test-slave-gkks krb5kdc[4025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517549, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:09.266436 4929 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:59:09.268033 4929 webserver.cc:460] Webserver started at http://127.22.26.195:34991/ using document root <none> and password file <none>
I20260927 13:59:09.268862 4929 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:09.269026 4929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:09.274670 4929 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 13:59:09.277616 4944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:09.278998 4929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 13:59:09.279203 4929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "ffd528ed51f646f7b3e4ace6220225f5"
format_stamp: "Formatted at 2026-09-27 13:59:06 on dist-test-slave-gkks"
server_key: "ad509d673eb06481ed044b83648f45a1"
server_key_iv: "44e96af4983826853556ed349bd3fba0"
server_key_version: "encryptionkey@0"
I20260927 13:59:09.280212 4929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:09.300386 4929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:09.304942 4929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:09.305220 4929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:09.306300 4929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:09.307716 4929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:09.307811 4929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:09.307924 4929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:09.308009 4929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:09.341420 4929 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:43269
I20260927 13:59:09.341614 5056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:43269 every 8 connection(s)
I20260927 13:59:09.342873 4929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:59:09.349184 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 4929
Sep 27 13:59:09 dist-test-slave-gkks krb5kdc[4025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517549, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:09.364732 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.350198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.22.26.254:39275)
0927 13:59:09.350998 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:09.351006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:09.351023 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:59:09.351116 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:09.351123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:09.351194 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:09.351302 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:09.352765 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.353537 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.354939 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.355654 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.355783 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.358515 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:09.358519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:09.358522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:09.358586 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:09.362081 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:09.362838 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:09.362841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:09.362843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:09.362955 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:09.363365 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:09.363369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:09.363372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:09.363673 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:09.363939 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:09.364211 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 13:59:09.364375 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"spinlock_wait_cycles":1792,"thread_start_us":564,"threads_started":1}
I20260927 13:59:09.365458 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.345529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:34329 (local address 127.22.26.254:39275)
0927 13:59:09.345744 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:09.345750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:09.346954 (+ 1204us) server_negotiation.cc:407] Connection header received
0927 13:59:09.348788 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:09.348793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:09.348860 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:09.348948 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:09.351029 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.351845 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.353389 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.353834 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.353969 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.357850 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:09.357855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:09.357859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:09.357889 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:09.361723 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:09.362524 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:09.362528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:09.362530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:09.362645 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:09.363270 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:09.363277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:09.363280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:09.363666 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:09.363835 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:09.364735 (+ 900us) server_negotiation.cc:299] Negotiation successful
0927 13:59:09.364945 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 13:59:09.366467 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.346122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:34329)
0927 13:59:09.346802 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:09.346840 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:09.348489 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:09.349241 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:09.349255 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:09.349955 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:59:09.350843 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:09.350876 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.352008 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.352014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:09.353230 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:09.353242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.354067 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.354075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:09.354313 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.355059 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:59:09.355117 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:59:09.357638 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:59:09.361853 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:09.361860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:09.361871 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:09.362410 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:09.362751 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:09.362757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:09.362763 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:09.362979 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:09.363825 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:59:09.363833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:59:09.364264 (+ 431us) client_negotiation.cc:769] Sending connection context
0927 13:59:09.364574 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 13:59:09.365200 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":264,"threads_started":1}
I20260927 13:59:09.368064 5057 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39275
I20260927 13:59:09.368415 5057 heartbeater.cc:464] Registering TS with master...
I20260927 13:59:09.369275 5057 heartbeater.cc:511] Master 127.22.26.254:39275 requested a full tablet report, sending...
I20260927 13:59:09.370831 4609 ts_manager.cc:194] Registered new tserver with Master: ffd528ed51f646f7b3e4ace6220225f5 (127.22.26.195:43269)
I20260927 13:59:09.371752 4609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:34329
I20260927 13:59:09.379856 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:59:09.386973 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.380780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.22.26.254:39275)
0927 13:59:09.381011 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:09.381020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:09.381397 (+ 377us) server_negotiation.cc:407] Connection header received
0927 13:59:09.381584 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:09.381588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:09.381719 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:09.381846 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:59:09.382280 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.383281 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.384826 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.385337 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.385435 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.385901 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:59:09.386398 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:59:09.386506 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:09.386610 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 13:59:09.386679 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:59:09.387910 4609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:59:09.395643 4609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:59:09.423384 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.408035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:33351 (local address 127.0.0.1:37322)
0927 13:59:09.408883 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:09.408926 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:09.409114 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:09.410235 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:09.410242 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:09.410268 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:09.410632 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:09.410646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.414313 (+ 3667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.414319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:09.416724 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:09.416739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.419897 (+ 3158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.419905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:09.420353 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.420509 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 13:59:09.420687 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 13:59:09.422700 (+ 2013us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":412,"threads_started":1}
I20260927 13:59:09.423645 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.408280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.22.26.194:33351)
0927 13:59:09.409534 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:09.409545 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:09.409576 (+ 31us) server_negotiation.cc:407] Connection header received
0927 13:59:09.409682 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:09.409693 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:09.409998 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:09.410831 (+ 833us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:09.412529 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.414144 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.417988 (+ 3844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.419542 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.422802 (+ 3260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.423008 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:09.423117 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 13:59:09.423306 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":895,"threads_started":1}
I20260927 13:59:09.427717 4861 tablet_service.cc:1467] Processing CreateTablet for tablet 3322a9466f344d709073e07b14c5aad9 (DEFAULT_TABLE table=test-table [id=db0150a2430b415789637df96635e8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:59:09.429101 4861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3322a9466f344d709073e07b14c5aad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:09.431362 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.410715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.22.26.195:43269)
0927 13:59:09.421588 (+ 10873us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:09.421601 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:09.421646 (+ 45us) server_negotiation.cc:407] Connection header received
0927 13:59:09.421770 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:09.421782 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:09.422117 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:09.424526 (+ 2409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:09.424807 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.426364 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.428760 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.430128 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.430284 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.430556 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:09.430844 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 13:59:09.430994 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10624,"thread_start_us":13580,"threads_started":1}
I20260927 13:59:09.431447 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.410071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:43269 (local address 127.0.0.1:51248)
0927 13:59:09.420473 (+ 10402us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:09.420499 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:09.420648 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:09.423934 (+ 3286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:09.423938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:09.423965 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:09.424309 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:09.424321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.426530 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.426535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:09.428582 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:09.428596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.430359 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.430365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:09.430654 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.430682 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 13:59:09.430800 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 13:59:09.430938 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10201,"mutex_wait_us":75,"thread_start_us":10674,"threads_started":1}
I20260927 13:59:09.435297 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.409438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35349 (local address 127.0.0.1:57388)
0927 13:59:09.409987 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:09.410005 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:09.410128 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:09.427712 (+ 17584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:09.427717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:09.427745 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:09.428041 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:09.428048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.429936 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.429942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:09.432604 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:09.432620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.434449 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.434455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:09.434629 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.434663 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 13:59:09.434762 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 13:59:09.434900 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":1,"thread_start_us":285,"threads_started":1}
I20260927 13:59:09.434854 4992 tablet_service.cc:1467] Processing CreateTablet for tablet 3322a9466f344d709073e07b14c5aad9 (DEFAULT_TABLE table=test-table [id=db0150a2430b415789637df96635e8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:59:09.435591 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.409751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.22.26.193:35349)
0927 13:59:09.422777 (+ 13026us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:09.422787 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:09.422813 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:59:09.422908 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:09.422916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:09.423195 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:09.423386 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:09.428232 (+ 4846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.429785 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.432795 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.434213 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.434361 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.434573 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:09.434830 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 13:59:09.434994 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12863,"thread_start_us":12727,"threads_started":1}
I20260927 13:59:09.436285 4992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3322a9466f344d709073e07b14c5aad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:09.439289 4729 tablet_service.cc:1467] Processing CreateTablet for tablet 3322a9466f344d709073e07b14c5aad9 (DEFAULT_TABLE table=test-table [id=db0150a2430b415789637df96635e8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:59:09.440645 4729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3322a9466f344d709073e07b14c5aad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:09.443344 5073 tablet_bootstrap.cc:492] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb: Bootstrap starting.
I20260927 13:59:09.446298 5073 tablet_bootstrap.cc:654] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:09.447373 5073 log.cc:824] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:09.450524 5073 tablet_bootstrap.cc:492] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb: No bootstrap required, opened a new log
I20260927 13:59:09.450824 5073 ts_tablet_manager.cc:1397] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 13:59:09.451635 5075 tablet_bootstrap.cc:492] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5: Bootstrap starting.
I20260927 13:59:09.453830 4794 heartbeater.cc:503] Master 127.22.26.254:39275 was elected leader, sending a full tablet report...
I20260927 13:59:09.455626 5076 tablet_bootstrap.cc:492] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: Bootstrap starting.
I20260927 13:59:09.456624 5075 tablet_bootstrap.cc:654] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:09.457885 5075 log.cc:824] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:09.458717 5076 tablet_bootstrap.cc:654] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:09.459991 5076 log.cc:824] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:09.461198 5075 tablet_bootstrap.cc:492] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5: No bootstrap required, opened a new log
I20260927 13:59:09.461493 5075 ts_tablet_manager.cc:1397] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 13:59:09.461828 5073 raft_consensus.cc:348] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:09.462208 5073 raft_consensus.cc:374] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:09.462309 5073 raft_consensus.cc:729] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3560ee761ed14470aefdb99eecf409eb, State: Initialized, Role: FOLLOWER
I20260927 13:59:09.463054 5076 tablet_bootstrap.cc:492] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: No bootstrap required, opened a new log
I20260927 13:59:09.462930 5073 consensus_queue.cc:260] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:09.463351 5076 ts_tablet_manager.cc:1397] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 13:59:09.464128 4926 heartbeater.cc:503] Master 127.22.26.254:39275 was elected leader, sending a full tablet report...
I20260927 13:59:09.464835 5073 ts_tablet_manager.cc:1428] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260927 13:59:09.472519 5075 raft_consensus.cc:348] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:09.473002 5075 raft_consensus.cc:374] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:09.473122 5075 raft_consensus.cc:729] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd528ed51f646f7b3e4ace6220225f5, State: Initialized, Role: FOLLOWER
I20260927 13:59:09.473896 5075 consensus_queue.cc:260] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:09.473963 5076 raft_consensus.cc:348] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:09.474363 5076 raft_consensus.cc:374] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:09.474499 5076 raft_consensus.cc:729] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee9f5e541a24f37bf449bc1cd141a28, State: Initialized, Role: FOLLOWER
I20260927 13:59:09.475194 5057 heartbeater.cc:503] Master 127.22.26.254:39275 was elected leader, sending a full tablet report...
I20260927 13:59:09.475101 5076 consensus_queue.cc:260] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:09.475876 5075 ts_tablet_manager.cc:1428] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260927 13:59:09.476835 5076 ts_tablet_manager.cc:1428] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
W20260927 13:59:09.594693 5058 tablet.cc:2369] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:59:09.640954 4927 tablet.cc:2369] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:59:09.668453 4795 tablet.cc:2369] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:59:09.668851 5081 raft_consensus.cc:482] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:59:09.669176 5081 raft_consensus.cc:504] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:09.670993 5081 leader_election.cc:288] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3560ee761ed14470aefdb99eecf409eb (127.22.26.194:33351), ffd528ed51f646f7b3e4ace6220225f5 (127.22.26.195:43269)
I20260927 13:59:09.677752 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.671566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:49499 (local address 127.22.26.194:33351)
0927 13:59:09.671811 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:09.671816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:09.672414 (+ 598us) server_negotiation.cc:407] Connection header received
0927 13:59:09.672691 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:09.672697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:09.672811 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:09.672942 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:09.673530 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.674232 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.676060 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.676821 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.677010 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.677099 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:09.677356 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 13:59:09.677484 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 13:59:09.677904 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.671566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:33351 (local address 127.22.26.193:49499)
0927 13:59:09.672297 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:09.672319 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:09.672464 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:09.672969 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:09.672974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:09.673016 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:09.673369 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:09.673379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.674364 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.674372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:09.675899 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:09.675911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.677028 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.677034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:09.677196 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.677220 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 13:59:09.677328 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 13:59:09.677458 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":1,"thread_start_us":410,"threads_started":1}
I20260927 13:59:09.678409 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.672650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:34645 (local address 127.22.26.195:43269)
0927 13:59:09.672885 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:09.672891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:09.673285 (+ 394us) server_negotiation.cc:407] Connection header received
0927 13:59:09.673496 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:09.673502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:09.673567 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:09.673663 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:09.674176 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.674888 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.676623 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.677412 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.677602 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.677678 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:09.678008 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 13:59:09.678163 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:59:09.678956 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.672486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:43269 (local address 127.22.26.193:34645)
0927 13:59:09.673177 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:09.673191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:09.673354 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:09.673681 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:09.673685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:09.673710 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:09.674016 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:09.674025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.675030 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.675036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:09.676464 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:09.676476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.677631 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.677638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:09.677792 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.677836 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 13:59:09.678595 (+ 759us) client_negotiation.cc:240] Negotiation successful
0927 13:59:09.678726 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":401,"threads_started":1}
I20260927 13:59:09.678968 4881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3322a9466f344d709073e07b14c5aad9" candidate_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3560ee761ed14470aefdb99eecf409eb" is_pre_election: true
I20260927 13:59:09.679589 4881 raft_consensus.cc:2454] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ee9f5e541a24f37bf449bc1cd141a28 in term 0.
I20260927 13:59:09.680069 5012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3322a9466f344d709073e07b14c5aad9" candidate_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd528ed51f646f7b3e4ace6220225f5" is_pre_election: true
I20260927 13:59:09.680315 4684 leader_election.cc:302] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3560ee761ed14470aefdb99eecf409eb, 6ee9f5e541a24f37bf449bc1cd141a28; no voters:
I20260927 13:59:09.680619 5012 raft_consensus.cc:2454] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ee9f5e541a24f37bf449bc1cd141a28 in term 0.
I20260927 13:59:09.680742 5081 raft_consensus.cc:2781] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:59:09.680883 5081 raft_consensus.cc:482] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:59:09.681012 5081 raft_consensus.cc:3037] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:09.683256 5081 raft_consensus.cc:504] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:09.683804 5081 leader_election.cc:288] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [CANDIDATE]: Term 1 election: Requested vote from peers 3560ee761ed14470aefdb99eecf409eb (127.22.26.194:33351), ffd528ed51f646f7b3e4ace6220225f5 (127.22.26.195:43269)
I20260927 13:59:09.684340 5012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3322a9466f344d709073e07b14c5aad9" candidate_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd528ed51f646f7b3e4ace6220225f5"
I20260927 13:59:09.684340 4881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3322a9466f344d709073e07b14c5aad9" candidate_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3560ee761ed14470aefdb99eecf409eb"
I20260927 13:59:09.684549 5012 raft_consensus.cc:3037] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:09.684540 4881 raft_consensus.cc:3037] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:09.687754 4881 raft_consensus.cc:2454] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ee9f5e541a24f37bf449bc1cd141a28 in term 1.
I20260927 13:59:09.687755 5012 raft_consensus.cc:2454] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ee9f5e541a24f37bf449bc1cd141a28 in term 1.
I20260927 13:59:09.688130 4684 leader_election.cc:302] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ee9f5e541a24f37bf449bc1cd141a28, ffd528ed51f646f7b3e4ace6220225f5; no voters:
I20260927 13:59:09.688516 5081 raft_consensus.cc:2781] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:59:09.689086 5081 raft_consensus.cc:686] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 1 LEADER]: Becoming Leader. State: Replica: 6ee9f5e541a24f37bf449bc1cd141a28, State: Running, Role: LEADER
I20260927 13:59:09.689699 5081 consensus_queue.cc:237] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } } peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } }
I20260927 13:59:09.694330 4609 catalog_manager.cc:5495] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ee9f5e541a24f37bf449bc1cd141a28 (127.22.26.193). New cstate: current_term: 1 leader_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ee9f5e541a24f37bf449bc1cd141a28" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 } health_report { overall_health: UNKNOWN } } }
I20260927 13:59:09.714874 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.709780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.22.26.193:35349)
0927 13:59:09.709973 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:09.709978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:09.709994 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:59:09.710138 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:09.710141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:09.710218 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:09.710335 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:59:09.710660 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.711442 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.712769 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.713159 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.713288 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.713801 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:59:09.714279 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:59:09.714435 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:09.714512 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 13:59:09.714588 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:59:09.722910 5012 raft_consensus.cc:1261] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5 [term 1 FOLLOWER]: Refusing update from remote peer 6ee9f5e541a24f37bf449bc1cd141a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:59:09.722941 4881 raft_consensus.cc:1261] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb [term 1 FOLLOWER]: Refusing update from remote peer 6ee9f5e541a24f37bf449bc1cd141a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:59:09.723877 5081 consensus_queue.cc:1035] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffd528ed51f646f7b3e4ace6220225f5" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:59:09.724197 5084 consensus_queue.cc:1035] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3560ee761ed14470aefdb99eecf409eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:59:09.743212 5087 mvcc.cc:205] Tried to move back new op lower bound from 7333959883655106560 to 7333959883533193216. Current Snapshot: MvccSnapshot[applied={T|T < 7333959883655106560}]
I20260927 13:59:09.754390 4608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:59:09.754700 4608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:59:09.758778 4608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Sending DeleteTablet for 3 replicas of tablet 3322a9466f344d709073e07b14c5aad9
I20260927 13:59:09.760049 4861 tablet_service.cc:1514] Processing DeleteTablet for tablet 3322a9466f344d709073e07b14c5aad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:59:09 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:37322
I20260927 13:59:09.760227 4992 tablet_service.cc:1514] Processing DeleteTablet for tablet 3322a9466f344d709073e07b14c5aad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:59:09 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:51248
I20260927 13:59:09.760375 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 4646
I20260927 13:59:09.761044 5097 tablet_replica.cc:330] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb: stopping tablet replica
I20260927 13:59:09.761447 5097 raft_consensus.cc:2229] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:59:09.761916 5097 raft_consensus.cc:2258] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:59:09.763805 5097 ts_tablet_manager.cc:1905] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:59:09.765667 5098 tablet_replica.cc:330] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5: stopping tablet replica
I20260927 13:59:09.766116 5098 raft_consensus.cc:2229] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:59:09.766660 5098 raft_consensus.cc:2258] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:59:09.766731 4729 tablet_service.cc:1514] Processing DeleteTablet for tablet 3322a9466f344d709073e07b14c5aad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:59:09 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:57388
I20260927 13:59:09.771219 5097 ts_tablet_manager.cc:1918] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:59:09.771416 5097 log.cc:1196] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/3322a9466f344d709073e07b14c5aad9
I20260927 13:59:09.772096 5097 ts_tablet_manager.cc:1939] T 3322a9466f344d709073e07b14c5aad9 P 3560ee761ed14470aefdb99eecf409eb: Deleting consensus metadata
I20260927 13:59:09.774430 4595 catalog_manager.cc:4841] TS 3560ee761ed14470aefdb99eecf409eb (127.22.26.194:33351): tablet 3322a9466f344d709073e07b14c5aad9 (table test-table [id=db0150a2430b415789637df96635e8a3]) successfully deleted
I20260927 13:59:09.774601 5098 ts_tablet_manager.cc:1905] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:59:09.775619 5099 tablet_replica.cc:330] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: stopping tablet replica
I20260927 13:59:09.776103 5099 raft_consensus.cc:2229] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:59:09.777213 5099 raft_consensus.cc:2258] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:59:09.779394 5099 ts_tablet_manager.cc:1905] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:59:09.781731 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.762151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57410 (local address 127.22.26.193:35349)
0927 13:59:09.762376 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:09.762384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:09.762407 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:59:09.762497 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:09.762504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:09.762604 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:09.762724 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:09.765692 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.767132 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.769499 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.770212 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.770359 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.773948 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:09.773972 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:09.773989 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:09.774041 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:09.778302 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:09.779277 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:09.779286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:09.779290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:09.779460 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:09.779967 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:09.779975 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:09.779981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:09.780442 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:09.780766 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:09.781076 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 13:59:09.781332 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:59:09.782691 4789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:09.786334 5098 ts_tablet_manager.cc:1918] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:59:09.786514 5098 log.cc:1196] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/3322a9466f344d709073e07b14c5aad9
I20260927 13:59:09.787104 5098 ts_tablet_manager.cc:1939] T 3322a9466f344d709073e07b14c5aad9 P ffd528ed51f646f7b3e4ace6220225f5: Deleting consensus metadata
I20260927 13:59:09.788604 4595 catalog_manager.cc:4841] TS ffd528ed51f646f7b3e4ace6220225f5 (127.22.26.195:43269): tablet 3322a9466f344d709073e07b14c5aad9 (table test-table [id=db0150a2430b415789637df96635e8a3]) successfully deleted
I20260927 13:59:09.883698 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 4646
W20260927 13:59:09.903051 4595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.26.193:35349 (error 108)
I20260927 13:59:09.903745 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 4798
I20260927 13:59:09.904958 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.904353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35349 (local address 127.0.0.1:57418)
0927 13:59:09.904638 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:09.904779 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260927 13:59:09.905232 4595 catalog_manager.cc:4584] TS 6ee9f5e541a24f37bf449bc1cd141a28 (127.22.26.193:35349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3322a9466f344d709073e07b14c5aad9: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35349: connect: Connection refused (error 111)
I20260927 13:59:09.918314 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.904511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.22.26.194:33351)
0927 13:59:09.904702 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:09.904710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:09.904780 (+ 70us) server_negotiation.cc:407] Connection header received
0927 13:59:09.904982 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:09.904988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:09.905069 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:09.905166 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:09.906632 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.907858 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.909224 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:09.909635 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:09.909734 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:09.912514 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:09.912525 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:09.912541 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:09.912583 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:09.915728 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:09.916453 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:09.916459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:09.916462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:09.916589 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:09.916936 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:09.916939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:09.916941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:09.917299 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:09.917508 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:09.917769 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 13:59:09.917933 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:59:09.918978 4921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:09.959052 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.958586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35349 (local address 127.0.0.1:57420)
0927 13:59:09.958789 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:09.958897 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 13:59:09.960341 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:09.959885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35349 (local address 127.0.0.1:57424)
0927 13:59:09.960134 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:09.960227 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260927 13:59:10.011358 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 4798
I20260927 13:59:10.021073 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:10.020498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35349 (local address 127.0.0.1:57440)
0927 13:59:10.020762 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:10.020879 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 13:59:10.022421 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:10.022008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35349 (local address 127.0.0.1:57454)
0927 13:59:10.022199 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:10.022289 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 13:59:10.032227 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 4929
I20260927 13:59:10.046299 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:10.033074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51252 (local address 127.22.26.195:43269)
0927 13:59:10.033281 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:10.033286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:10.033323 (+ 37us) server_negotiation.cc:407] Connection header received
0927 13:59:10.033587 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:10.033591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:10.033680 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:10.033792 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:10.035188 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:10.035994 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:10.037370 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:10.037766 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:10.037901 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:10.040766 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:10.040781 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:10.040797 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:10.040830 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:10.043547 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:10.044258 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:10.044265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:10.044268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:10.044407 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:10.044782 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:10.044788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:10.044791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:10.045142 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:10.045436 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:10.045744 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 13:59:10.045971 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:59:10.046969 5052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:10.103341 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:10.102772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35349 (local address 127.0.0.1:57460)
0927 13:59:10.103000 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:10.103160 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 13:59:10.104380 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:10.103991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35349 (local address 127.0.0.1:57462)
0927 13:59:10.104193 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:10.104262 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 13:59:10.135962 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 4929
I20260927 13:59:10.152695 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 4576
I20260927 13:59:10.154016 4639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:10.271601 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 4576
I20260927 13:59:10.295640 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39275
--webserver_interface=127.22.26.254
--webserver_port=45423
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:39275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:10.627344 5105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:10.627637 5105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:10.627781 5105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:10.635361 5105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:59:10.635473 5105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:10.635555 5105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:10.635655 5105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:59:10.635731 5105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:59:10.645555 5105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:39275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=45423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:10.646976 5105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:10.648386 5105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:10.660270 5110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:10.660240 5113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:10.660342 5111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:10.661010 5105 server_base.cc:1034] running on GCE node
I20260927 13:59:10.661844 5105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:10.663337 5105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:10.664569 5105 hybrid_clock.cc:648] HybridClock initialized: now 1790517550664518 us; error 56 us; skew 500 ppm
I20260927 13:59:10.664968 5105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:10 dist-test-slave-gkks krb5kdc[4025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517550, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:10.668112 5105 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:59:10.669586 5105 webserver.cc:460] Webserver started at http://127.22.26.254:45423/ using document root <none> and password file <none>
I20260927 13:59:10.670231 5105 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:10.670336 5105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:10.675856 5105 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:59:10.678602 5120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:10.679962 5105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:59:10.680102 5105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "ad383ba76dfe42f39684e684dc2bf160"
format_stamp: "Formatted at 2026-09-27 13:59:04 on dist-test-slave-gkks"
server_key: "2b2f923b170661793b0ea5944a4ea579"
server_key_iv: "e3b5b7b42c08f77334f1cf6ff8497c61"
server_key_version: "encryptionkey@0"
I20260927 13:59:10.681026 5105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:10.717826 5105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:10.721848 5105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:10.722064 5105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:10.740933 5105 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:39275
I20260927 13:59:10.740952 5171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:39275 every 8 connection(s)
I20260927 13:59:10.742446 5105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:59:10.749380 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 5105
I20260927 13:59:10.749644 5172 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:59:10.750414 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:35349
--local_ip_for_outbound_sockets=127.22.26.193
--tserver_master_addrs=127.22.26.254:39275
--webserver_port=38521
--webserver_interface=127.22.26.193
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:59:10.754835 5172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Bootstrap starting.
I20260927 13:59:10.776712 5172 log.cc:824] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:10.793406 5172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:59:10.793975 5172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160: Bootstrap complete.
I20260927 13:59:10.803071 5172 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } }
I20260927 13:59:10.803987 5172 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad383ba76dfe42f39684e684dc2bf160, State: Initialized, Role: FOLLOWER
I20260927 13:59:10.804610 5172 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } }
I20260927 13:59:10.804834 5172 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:59:10.804946 5172 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:59:10.805163 5172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 2 FOLLOWER]: Advancing to term 3
I20260927 13:59:10.809010 5172 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } }
I20260927 13:59:10.809537 5172 leader_election.cc:302] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad383ba76dfe42f39684e684dc2bf160; no voters:
I20260927 13:59:10.810237 5172 leader_election.cc:288] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 13:59:10.810387 5176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 13:59:10.811446 5176 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [term 3 LEADER]: Becoming Leader. State: Replica: ad383ba76dfe42f39684e684dc2bf160, State: Running, Role: LEADER
I20260927 13:59:10.812080 5176 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } }
I20260927 13:59:10.812448 5172 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:59:10.814451 5178 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad383ba76dfe42f39684e684dc2bf160. Latest consensus state: current_term: 3 leader_uuid: "ad383ba76dfe42f39684e684dc2bf160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } } }
I20260927 13:59:10.814833 5178 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: This master's current role is: LEADER
I20260927 13:59:10.815254 5177 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ad383ba76dfe42f39684e684dc2bf160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad383ba76dfe42f39684e684dc2bf160" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39275 } } }
I20260927 13:59:10.815510 5177 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad383ba76dfe42f39684e684dc2bf160 [sys.catalog]: This master's current role is: LEADER
I20260927 13:59:10.817441 5183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:59:10.828235 5183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:59:10.830256 5183 catalog_manager.cc:1259] Loaded cluster ID: 48f19a64b8df495fa62253deb3b1f699
I20260927 13:59:10.830442 5183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 13:59:10.833138 5183 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc3c4e799ad google::LogMessage::Fail() at ??:0
@ 0x7fc3c4e7c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc3c4e794de google::LogMessage::Flush() at ??:0
@ 0x7fc3c4e7aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc3da8b0512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fc3da8ad976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc3da935610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fc3da935439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc3c7ac5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc3c7ac891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fc3c7ac8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc3c7aa3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc3cad7d6db start_thread at ??:0
@ 0x7fc3c3d1361f clone at ??:0
W20260927 13:59:11.141103 5174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:11.141522 5174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:11.141671 5174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:11.151002 5174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:11.151171 5174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:11.151377 5174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:59:11.162665 5174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:35349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=38521
--tserver_master_addrs=127.22.26.254:39275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:11.164192 5174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:11.165551 5174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:11.178844 5213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:11.179064 5216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:11.178910 5214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:11.179723 5174 server_base.cc:1034] running on GCE node
I20260927 13:59:11.180275 5174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:11.181154 5174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:11.182385 5174 hybrid_clock.cc:648] HybridClock initialized: now 1790517551182333 us; error 34 us; skew 500 ppm
I20260927 13:59:11.182696 5174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:11 dist-test-slave-gkks krb5kdc[4025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517551, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:11.185860 5174 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:59:11.187494 5174 webserver.cc:460] Webserver started at http://127.22.26.193:38521/ using document root <none> and password file <none>
I20260927 13:59:11.188181 5174 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:11.188287 5174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:11.193605 5174 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 13:59:11.198300 5223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:11.199821 5174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:59:11.199986 5174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "6ee9f5e541a24f37bf449bc1cd141a28"
format_stamp: "Formatted at 2026-09-27 13:59:05 on dist-test-slave-gkks"
server_key: "e92a4233c63e8cba329614408d10e2b6"
server_key_iv: "c30fdff0cfedaf2b8f2ffffde0acca57"
server_key_version: "encryptionkey@0"
I20260927 13:59:11.200914 5174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:11.217780 5174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:11.221823 5174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:11.222059 5174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:11.223008 5174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:11.224817 5230 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260927 13:59:11.231225 5230 ts_tablet_manager.cc:1852] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 13:59:11.232246 5230 ts_tablet_manager.cc:1905] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:59:11.240127 5230 ts_tablet_manager.cc:1918] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:59:11.240360 5230 log.cc:1196] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/3322a9466f344d709073e07b14c5aad9
I20260927 13:59:11.240837 5230 ts_tablet_manager.cc:1939] T 3322a9466f344d709073e07b14c5aad9 P 6ee9f5e541a24f37bf449bc1cd141a28: Deleting consensus metadata
I20260927 13:59:11.241268 5174 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 13:59:11.241441 5174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260927 13:59:11.241601 5174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:11.241696 5174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:11.276508 5174 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:35349
I20260927 13:59:11.276623 5336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:35349 every 8 connection(s)
I20260927 13:59:11.277946 5174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:59:11.283047 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:11.281197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:36303)
0927 13:59:11.281916 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:11.282177 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":271,"threads_started":1}
W20260927 13:59:11.283761 5337 heartbeater.cc:644] Failed to heartbeat to 127.22.26.254:39275 (0 consecutive failures): Network error: Failed to ping master at 127.22.26.254:39275: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
I20260927 13:59:11.284579 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:11.284276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:34617)
0927 13:59:11.284409 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:11.284480 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 13:59:11.285416 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:11.285116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:60577)
0927 13:59:11.285234 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:11.285304 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 13:59:11.285692 5337 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 13:59:11.287992 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 5174
I20260927 13:59:11.288645 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:33351
--local_ip_for_outbound_sockets=127.22.26.194
--tserver_master_addrs=127.22.26.254:39275
--webserver_port=46149
--webserver_interface=127.22.26.194
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:11.621763 5340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:11.622052 5340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:11.622217 5340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:11.629482 5340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:11.629601 5340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:11.629794 5340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:59:11.639694 5340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:33351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=46149
--tserver_master_addrs=127.22.26.254:39275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:11.641036 5340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:11.642364 5340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:11.655822 5345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:11.655764 5348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:11.655786 5346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:11.656622 5340 server_base.cc:1034] running on GCE node
I20260927 13:59:11.657276 5340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:11.658206 5340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:11.659466 5340 hybrid_clock.cc:648] HybridClock initialized: now 1790517551659418 us; error 45 us; skew 500 ppm
I20260927 13:59:11.659881 5340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:11 dist-test-slave-gkks krb5kdc[4025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517551, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:11.663213 5340 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:59:11.664702 5340 webserver.cc:460] Webserver started at http://127.22.26.194:46149/ using document root <none> and password file <none>
I20260927 13:59:11.665403 5340 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:11.665566 5340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:11.671386 5340 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:59:11.674204 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:11.675711 5340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:59:11.675897 5340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "3560ee761ed14470aefdb99eecf409eb"
format_stamp: "Formatted at 2026-09-27 13:59:06 on dist-test-slave-gkks"
server_key: "159d44b6607bd2b85757eea0701994c3"
server_key_iv: "65db2cd4498faa8594069dda63a97ae9"
server_key_version: "encryptionkey@0"
I20260927 13:59:11.676911 5340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:11.700763 5340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:11.705396 5340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:11.705752 5340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:11.706912 5340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:11.708385 5340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:11.708477 5340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:11.708559 5340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:11.708619 5340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:11.742336 5340 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:33351
I20260927 13:59:11.742391 5467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:33351 every 8 connection(s)
I20260927 13:59:11.743854 5340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:59:11.748644 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:11.746788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:36371)
0927 13:59:11.747555 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:11.747781 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":317,"threads_started":1}
W20260927 13:59:11.749428 5468 heartbeater.cc:644] Failed to heartbeat to 127.22.26.254:39275 (0 consecutive failures): Network error: Failed to ping master at 127.22.26.254:39275: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
I20260927 13:59:11.750382 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:11.750023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:53297)
0927 13:59:11.750197 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:11.750273 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 13:59:11.751312 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:11.750934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:54709)
0927 13:59:11.751108 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:11.751191 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 13:59:11.751636 5468 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 13:59:11.751866 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 5340
I20260927 13:59:11.752566 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:43269
--local_ip_for_outbound_sockets=127.22.26.195
--tserver_master_addrs=127.22.26.254:39275
--webserver_port=34991
--webserver_interface=127.22.26.195
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:12.085917 5471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:12.086212 5471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:12.086372 5471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:12.093626 5471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:12.093743 5471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:12.093921 5471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:59:12.103734 5471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:43269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=34991
--tserver_master_addrs=127.22.26.254:39275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:12.105216 5471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:12.106549 5471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:12.119915 5476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:12.119832 5479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:12.119860 5477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:12.120754 5471 server_base.cc:1034] running on GCE node
I20260927 13:59:12.121418 5471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:12.122376 5471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:12.123605 5471 hybrid_clock.cc:648] HybridClock initialized: now 1790517552123540 us; error 54 us; skew 500 ppm
I20260927 13:59:12.123927 5471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:12 dist-test-slave-gkks krb5kdc[4025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517552, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:12.126998 5471 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:59:12.128531 5471 webserver.cc:460] Webserver started at http://127.22.26.195:34991/ using document root <none> and password file <none>
I20260927 13:59:12.129221 5471 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:12.129374 5471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:12.134611 5471 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:59:12.137434 5486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:12.138744 5471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 13:59:12.138913 5471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "ffd528ed51f646f7b3e4ace6220225f5"
format_stamp: "Formatted at 2026-09-27 13:59:06 on dist-test-slave-gkks"
server_key: "ad509d673eb06481ed044b83648f45a1"
server_key_iv: "44e96af4983826853556ed349bd3fba0"
server_key_version: "encryptionkey@0"
I20260927 13:59:12.139935 5471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:12.157181 5471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:12.161180 5471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:12.161451 5471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:12.162369 5471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:12.163698 5471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:12.163820 5471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:12.163951 5471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:12.164040 5471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:12.197512 5471 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:43269
I20260927 13:59:12.197606 5598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:43269 every 8 connection(s)
I20260927 13:59:12.199005 5471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:59:12.203694 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:12.201870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:43229)
0927 13:59:12.202597 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:12.202809 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":272,"threads_started":1}
W20260927 13:59:12.204504 5599 heartbeater.cc:644] Failed to heartbeat to 127.22.26.254:39275 (0 consecutive failures): Network error: Failed to ping master at 127.22.26.254:39275: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
I20260927 13:59:12.205297 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:12.204987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:36253)
0927 13:59:12.205118 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:12.205181 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 13:59:12.206225 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:12.205776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:59197)
0927 13:59:12.205952 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:12.206072 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 13:59:12.206321 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 5471
W20260927 13:59:12.206490 5599 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 13:59:12.287043 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:12.286203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:39709)
0927 13:59:12.286710 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:12.286794 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":280,"threads_started":1}
I20260927 13:59:12.752990 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:12.752229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:59569)
0927 13:59:12.752670 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:12.752767 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":209,"threads_started":1}
I20260927 13:59:13.207819 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:13.207004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:54047)
0927 13:59:13.207481 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:13.207595 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":259,"threads_started":1}
I20260927 13:59:13.288730 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:13.288023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:59157)
0927 13:59:13.288461 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:13.288566 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":236,"threads_started":1}
I20260927 13:59:13.754644 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:13.753909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:34987)
0927 13:59:13.754391 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:13.754481 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":258,"threads_started":1}
I20260927 13:59:14.209730 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:14.208735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:56079)
0927 13:59:14.209433 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:14.209549 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":399,"threads_started":1}
I20260927 13:59:14.290485 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:14.289680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:48607)
0927 13:59:14.290184 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:14.290273 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":253,"threads_started":1}
I20260927 13:59:14.756306 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:14.755626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:55477)
0927 13:59:14.756045 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:14.756126 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":221,"threads_started":1}
I20260927 13:59:15.211421 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:15.210652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:33487)
0927 13:59:15.211172 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:15.211257 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":245,"threads_started":1}
I20260927 13:59:15.292393 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:15.291567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:34659)
0927 13:59:15.292133 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:15.292221 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":300,"threads_started":1}
I20260927 13:59:15.758137 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:15.757277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:50949)
0927 13:59:15.757881 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:15.757962 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":305,"threads_started":1}
I20260927 13:59:16.213105 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:16.212336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:54105)
0927 13:59:16.212810 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:16.212895 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":225,"threads_started":1}
I20260927 13:59:16.294328 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:16.293553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:34429)
0927 13:59:16.294045 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:16.294134 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":245,"threads_started":1}
I20260927 13:59:16.760004 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:16.759238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:56339)
0927 13:59:16.759740 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:16.759826 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":266,"threads_started":1}
I20260927 13:59:17.214820 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:17.214057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:45247)
0927 13:59:17.214571 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:17.214651 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":263,"threads_started":1}
I20260927 13:59:17.296092 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:17.295411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:40715)
0927 13:59:17.295848 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:17.295928 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":215,"threads_started":1}
I20260927 13:59:17.761723 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:17.760972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:57939)
0927 13:59:17.761443 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:17.761559 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":264,"threads_started":1}
I20260927 13:59:18.216372 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:18.215712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:36811)
0927 13:59:18.216129 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:18.216219 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":220,"threads_started":1}
I20260927 13:59:18.297832 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:18.297014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:57977)
0927 13:59:18.297540 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:18.297625 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":226,"threads_started":1}
I20260927 13:59:18.763629 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:18.762740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:43801)
0927 13:59:18.763336 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:18.763418 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":331,"threads_started":1}
I20260927 13:59:19.218094 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:19.217307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:47053)
0927 13:59:19.217790 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:19.217873 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":232,"threads_started":1}
I20260927 13:59:19.299789 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:19.298867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:43549)
0927 13:59:19.299496 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:19.299588 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":406,"threads_started":1}
I20260927 13:59:19.765286 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:19.764584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:44969)
0927 13:59:19.765041 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:19.765124 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
I20260927 13:59:20.219918 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:20.219217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:35451)
0927 13:59:20.219676 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:20.219757 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":234,"threads_started":1}
I20260927 13:59:20.301788 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:20.300874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:36027)
0927 13:59:20.301455 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:20.301587 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":312,"threads_started":1}
I20260927 13:59:20.767351 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:20.766344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:41209)
0927 13:59:20.766986 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:20.767094 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":369,"threads_started":1}
I20260927 13:59:21.221552 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:21.220797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:40599)
0927 13:59:21.221258 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:21.221344 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":221,"threads_started":1}
I20260927 13:59:21.303702 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:21.302834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:56003)
0927 13:59:21.303399 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:21.303528 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":336,"threads_started":1}
I20260927 13:59:21.769030 5631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:21.768322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:55085)
0927 13:59:21.768781 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:21.768864 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":257,"threads_started":1}
I20260927 13:59:22.223322 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:22.222586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:51657)
0927 13:59:22.223025 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:22.223164 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":232,"threads_started":1}
I20260927 13:59:22.305610 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:22.304679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:59691)
0927 13:59:22.305260 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:22.305372 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":281,"threads_started":1}
I20260927 13:59:22.770686 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:22.769979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:56843)
0927 13:59:22.770399 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:22.770500 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":204,"threads_started":1}
I20260927 13:59:23.225029 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:23.224305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:55433)
0927 13:59:23.224765 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:23.224852 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":258,"threads_started":1}
I20260927 13:59:23.307452 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:23.306567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:46463)
0927 13:59:23.307024 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:23.307275 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":239,"threads_started":1}
I20260927 13:59:23.772382 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:23.771706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:48133)
0927 13:59:23.772141 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:23.772223 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":213,"threads_started":1}
I20260927 13:59:24.226786 5638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:24.226065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:46639)
0927 13:59:24.226539 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:24.226632 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":238,"threads_started":1}
I20260927 13:59:24.309489 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:24.308534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:34231)
0927 13:59:24.309172 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:24.309281 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":363,"threads_started":1}
I20260927 13:59:24.774225 5640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:24.773405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:39997)
0927 13:59:24.773919 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:24.774021 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":242,"threads_started":1}
I20260927 13:59:25.228608 5641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:25.227832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:52123)
0927 13:59:25.228309 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:25.228426 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":233,"threads_started":1}
I20260927 13:59:25.311326 5642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:25.310543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:38455)
0927 13:59:25.311023 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:25.311138 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":223,"threads_started":1}
I20260927 13:59:25.775990 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:25.775246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:34111)
0927 13:59:25.775692 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:25.775792 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":215,"threads_started":1}
I20260927 13:59:26.230247 5644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:26.229514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:53449)
0927 13:59:26.229985 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:26.230075 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260927 13:59:26.313413 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:26.312599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:40797)
0927 13:59:26.313123 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:26.313220 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":244,"threads_started":1}
I20260927 13:59:26.777797 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:26.777073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.194:46935)
0927 13:59:26.777511 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:26.777618 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":222,"threads_started":1}
I20260927 13:59:27.211864 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 5174
I20260927 13:59:27.231554 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:27.213130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.22.26.193:35349)
0927 13:59:27.213828 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:27.213849 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:27.213897 (+ 48us) server_negotiation.cc:407] Connection header received
0927 13:59:27.216024 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:27.216034 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:27.216494 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:27.216852 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:27.218381 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:27.219727 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:27.221031 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:27.221711 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:27.221920 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:27.224934 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:27.224942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:27.224958 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:27.224998 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:27.228283 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:27.228985 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:27.228994 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:27.229000 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:27.229163 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:27.229535 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:27.229540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:27.229543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:27.230061 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:27.230367 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:27.230803 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 13:59:27.231244 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":307,"threads_started":1}
I20260927 13:59:27.232287 5648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:27.231419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.195:50647)
0927 13:59:27.231949 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:27.232074 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"threads_started":1}
I20260927 13:59:27.233222 5332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:27.319321 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 5174
I20260927 13:59:27.319414 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:27.318225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39275 (local address 127.22.26.193:40283)
0927 13:59:27.318804 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:27.319039 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":301,"threads_started":1}
I20260927 13:59:27.337148 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 5340
I20260927 13:59:27.355579 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:27.338041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.22.26.194:33351)
0927 13:59:27.338572 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:27.338589 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:27.338630 (+ 41us) server_negotiation.cc:407] Connection header received
0927 13:59:27.340175 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:27.340185 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:27.340636 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:27.341010 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:27.342549 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:27.343977 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:27.345326 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:27.346027 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:27.346226 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:27.349074 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:27.349082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:27.349097 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:27.349132 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:27.352253 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:27.352965 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:27.352972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:27.352979 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:27.353148 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:27.353550 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:27.353557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:27.353564 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:27.354068 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:27.354373 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:27.354828 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 13:59:27.355283 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":227,"threads_started":1}
I20260927 13:59:27.357038 5463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:27.444802 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 5340
I20260927 13:59:27.462972 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 5471
I20260927 13:59:27.481334 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:27.464293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.22.26.195:43269)
0927 13:59:27.464881 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:27.464896 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:27.464940 (+ 44us) server_negotiation.cc:407] Connection header received
0927 13:59:27.466549 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:27.466557 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:27.466991 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:27.467397 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:27.468932 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:27.470123 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:27.471472 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:27.472110 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:27.472311 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:27.475293 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:27.475301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:27.475316 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:27.475351 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:27.477992 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:27.478712 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:27.478723 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:27.478736 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:27.478895 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:27.479381 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:27.479385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:27.479387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:27.479875 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:27.480198 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:27.480636 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 13:59:27.481047 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":269,"threads_started":1}
I20260927 13:59:27.482712 5594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:27.565726 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 5471
2026-09-27T13:59:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24442 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:59:27 dist-test-slave-gkks krb5kdc[5657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:59:27 dist-test-slave-gkks krb5kdc[5657](info): set up 2 sockets
Sep 27 13:59:27 dist-test-slave-gkks krb5kdc[5657](info): commencing operation
krb5kdc: starting...
W20260927 13:59:28.679997 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 13:59:28 dist-test-slave-gkks krb5kdc[5657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:59:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:59:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 13:59:28.935647 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42401
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:42401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:29.273767 5670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:29.274077 5670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:29.274169 5670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:29.282234 5670 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 13:59:29.282334 5670 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 13:59:29.282371 5670 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 13:59:29.282419 5670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:59:29.282473 5670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:29.282541 5670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:29.282591 5670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:59:29.282644 5670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:59:29.292954 5670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:42401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:29.294299 5670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:29.295744 5670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:29.308677 5675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:29.308740 5678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:29.308766 5676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:29.309808 5670 server_base.cc:1034] running on GCE node
I20260927 13:59:29.310742 5670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:29.312372 5670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:29.313630 5670 hybrid_clock.cc:648] HybridClock initialized: now 1790517569313571 us; error 41 us; skew 500 ppm
I20260927 13:59:29.313975 5670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:29 dist-test-slave-gkks krb5kdc[5657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517569, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:29.317392 5670 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:59:29.318903 5670 webserver.cc:460] Webserver started at http://127.22.26.254:46199/ using document root <none> and password file <none>
I20260927 13:59:29.319748 5670 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:29.319839 5670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:29.320207 5670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:59:29.323726 5670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "d803bc2147c048cf8be3d45fd18fb911"
format_stamp: "Formatted at 2026-09-27 13:59:29 on dist-test-slave-gkks"
server_key: "4efdc8b9c3f39a9fdde08ab6e7ed1217"
server_key_iv: "3579a200d8c4b6115c3af4b46a0026fe"
server_key_version: "encryptionkey@0"
I20260927 13:59:29.324679 5670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "d803bc2147c048cf8be3d45fd18fb911"
format_stamp: "Formatted at 2026-09-27 13:59:29 on dist-test-slave-gkks"
server_key: "4efdc8b9c3f39a9fdde08ab6e7ed1217"
server_key_iv: "3579a200d8c4b6115c3af4b46a0026fe"
server_key_version: "encryptionkey@0"
I20260927 13:59:29.330574 5670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 13:59:29.334458 5685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:29.336174 5670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:59:29.336393 5670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "d803bc2147c048cf8be3d45fd18fb911"
format_stamp: "Formatted at 2026-09-27 13:59:29 on dist-test-slave-gkks"
server_key: "4efdc8b9c3f39a9fdde08ab6e7ed1217"
server_key_iv: "3579a200d8c4b6115c3af4b46a0026fe"
server_key_version: "encryptionkey@0"
I20260927 13:59:29.336658 5670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:29.352866 5670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:29.357299 5670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:29.357601 5670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:29.377565 5670 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:42401
I20260927 13:59:29.377586 5736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:42401 every 8 connection(s)
I20260927 13:59:29.378985 5670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:59:29.380887 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 5670
I20260927 13:59:29.381147 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 13:59:29.382006 22635 external_mini_cluster.cc:1428] Setting key 64d7e293e9d9b0b5f7caa09ccdc7383d
I20260927 13:59:29.383917 5737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:59:29 dist-test-slave-gkks krb5kdc[5657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:29.398293 5737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Bootstrap starting.
I20260927 13:59:29.401592 5737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:29.402655 5737 log.cc:824] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:29.405664 5737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: No bootstrap required, opened a new log
I20260927 13:59:29.406250 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:29.384255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53700 (local address 127.22.26.254:42401)
0927 13:59:29.385103 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:29.385115 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:29.385157 (+ 42us) server_negotiation.cc:407] Connection header received
0927 13:59:29.386758 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:29.386779 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:29.387234 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:29.387581 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:29.389523 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:29.390944 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:29.392910 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:29.393589 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:29.393731 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:29.398691 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:29.398699 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:29.398713 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:29.398754 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:29.401939 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:29.402728 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:29.402734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:29.402740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:29.402944 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:29.403372 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:29.403377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:29.403380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:29.403928 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:29.404247 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:29.404717 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 13:59:29.405161 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":439,"threads_started":1}
I20260927 13:59:29.414120 5737 raft_consensus.cc:348] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } }
I20260927 13:59:29.414535 5737 raft_consensus.cc:374] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:29.414652 5737 raft_consensus.cc:729] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d803bc2147c048cf8be3d45fd18fb911, State: Initialized, Role: FOLLOWER
I20260927 13:59:29.415273 5737 consensus_queue.cc:260] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } }
I20260927 13:59:29.415527 5737 raft_consensus.cc:388] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:59:29.415673 5737 raft_consensus.cc:482] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:59:29.415863 5737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:29.418428 5737 raft_consensus.cc:504] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } }
I20260927 13:59:29.419004 5737 leader_election.cc:302] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d803bc2147c048cf8be3d45fd18fb911; no voters:
I20260927 13:59:29.419669 5737 leader_election.cc:288] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:59:29.419833 5742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:59:29.420532 5742 raft_consensus.cc:686] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 1 LEADER]: Becoming Leader. State: Replica: d803bc2147c048cf8be3d45fd18fb911, State: Running, Role: LEADER
I20260927 13:59:29.421262 5742 consensus_queue.cc:237] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } }
I20260927 13:59:29.423487 5742 sys_catalog.cc:455] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d803bc2147c048cf8be3d45fd18fb911. Latest consensus state: current_term: 1 leader_uuid: "d803bc2147c048cf8be3d45fd18fb911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } } }
I20260927 13:59:29.423884 5742 sys_catalog.cc:458] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: This master's current role is: LEADER
I20260927 13:59:29.425027 5744 sys_catalog.cc:455] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d803bc2147c048cf8be3d45fd18fb911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } } }
I20260927 13:59:29.425347 5744 sys_catalog.cc:458] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: This master's current role is: LEADER
I20260927 13:59:29.428471 5737 sys_catalog.cc:564] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:59:29.433069 5746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:59:29.438625 5746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:59:29.448398 5746 catalog_manager.cc:1347] Generated new cluster ID: 4cab217f3b4e43bcb9b65e13c688fd44
I20260927 13:59:29.448529 5746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:59:29.465452 5746 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:59:29.466859 5746 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:59:29.483961 5746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Generated new TSK 0
I20260927 13:59:29.484838 5746 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 13:59:29.555776 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:42401
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:29.919888 5763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:29.920176 5763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:29.920269 5763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:29.927518 5763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:29.927621 5763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:29.927733 5763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:59:29.938413 5763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:29.939739 5763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:29.940982 5763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:29.954483 5771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:29.954579 5768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:29.954555 5769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:29.955394 5763 server_base.cc:1034] running on GCE node
I20260927 13:59:29.955973 5763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:29.956892 5763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:29.958083 5763 hybrid_clock.cc:648] HybridClock initialized: now 1790517569958031 us; error 52 us; skew 500 ppm
I20260927 13:59:29.958386 5763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:29 dist-test-slave-gkks krb5kdc[5657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517569, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:29.961479 5763 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:59:29.962926 5763 webserver.cc:460] Webserver started at http://127.22.26.193:38635/ using document root <none> and password file <none>
I20260927 13:59:29.963649 5763 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:29.963779 5763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:29.964092 5763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:59:29.966867 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "3b3d36d518f14027ba4f7c458fe7b8fb"
format_stamp: "Formatted at 2026-09-27 13:59:29 on dist-test-slave-gkks"
server_key: "93efd19c5d88f27f12f4116f9c11f58c"
server_key_iv: "35ab40ae05d64831c3d14d7c53f45cd5"
server_key_version: "encryptionkey@0"
I20260927 13:59:29.967705 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "3b3d36d518f14027ba4f7c458fe7b8fb"
format_stamp: "Formatted at 2026-09-27 13:59:29 on dist-test-slave-gkks"
server_key: "93efd19c5d88f27f12f4116f9c11f58c"
server_key_iv: "35ab40ae05d64831c3d14d7c53f45cd5"
server_key_version: "encryptionkey@0"
I20260927 13:59:29.972606 5763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:59:29.976117 5778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:29.977308 5763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:59:29.977483 5763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "3b3d36d518f14027ba4f7c458fe7b8fb"
format_stamp: "Formatted at 2026-09-27 13:59:29 on dist-test-slave-gkks"
server_key: "93efd19c5d88f27f12f4116f9c11f58c"
server_key_iv: "35ab40ae05d64831c3d14d7c53f45cd5"
server_key_version: "encryptionkey@0"
I20260927 13:59:29.977710 5763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:29.989354 5763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:29.993729 5763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:29.994025 5763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:29.994971 5763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:29.996316 5763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:29.996421 5763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:29.996578 5763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:29.996654 5763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:30.031032 5763 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:33815
I20260927 13:59:30.031114 5890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:33815 every 8 connection(s)
I20260927 13:59:30.032394 5763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:59:30.034358 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 5763
I20260927 13:59:30.034602 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 13:59:30.035502 22635 external_mini_cluster.cc:1428] Setting key b9c5fbb677a2d85538de3b45b63bdfa6
Sep 27 13:59:30 dist-test-slave-gkks krb5kdc[5657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517569, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:30.054515 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:30.035161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:59281 (local address 127.22.26.254:42401)
0927 13:59:30.035676 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:30.035683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:30.036751 (+ 1068us) server_negotiation.cc:407] Connection header received
0927 13:59:30.038780 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:30.038789 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:30.038878 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:30.039007 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:30.040496 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:30.041759 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:30.043370 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:30.044069 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:30.044212 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:30.047946 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:30.047951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:30.047955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:30.047997 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:30.050857 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:30.051853 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:30.051858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:30.051860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:30.051996 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:30.052637 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:30.052641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:30.052644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:30.052947 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:30.053071 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:30.054044 (+ 973us) server_negotiation.cc:299] Negotiation successful
0927 13:59:30.054226 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":250,"threads_started":1}
I20260927 13:59:30.056034 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:30.035783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:59281)
0927 13:59:30.036591 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:30.036637 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:30.038491 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:30.039250 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:30.039258 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:30.039764 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:59:30.040338 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:30.040355 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:30.041932 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:30.041936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:30.043224 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:30.043240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:30.044281 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:30.044286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:30.044472 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:30.045180 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:59:30.045201 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:59:30.047692 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:59:30.051001 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:30.051012 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:30.051032 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:30.051714 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:30.052140 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:30.052148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:30.052154 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:30.052468 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:30.053094 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:59:30.053105 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:59:30.053570 (+ 465us) client_negotiation.cc:769] Sending connection context
0927 13:59:30.053885 (+ 315us) client_negotiation.cc:240] Negotiation successful
0927 13:59:30.054484 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":337,"threads_started":1}
I20260927 13:59:30.057972 5891 heartbeater.cc:347] Connected to a master server at 127.22.26.254:42401
I20260927 13:59:30.058319 5891 heartbeater.cc:464] Registering TS with master...
I20260927 13:59:30.059129 5891 heartbeater.cc:511] Master 127.22.26.254:42401 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
I20260927 13:59:30.061592 5702 ts_manager.cc:194] Registered new tserver with Master: 3b3d36d518f14027ba4f7c458fe7b8fb (127.22.26.193:33815)
I20260927 13:59:30.063382 5702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:59281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 13:59:30.095183 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:42401
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:30.421308 5897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:30.421571 5897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:30.421667 5897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:30.428735 5897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:30.428869 5897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:30.429011 5897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:59:30.438887 5897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:30.440145 5897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:30.441547 5897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:30.454736 5902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:30.454780 5905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:30.454802 5903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:30.455590 5897 server_base.cc:1034] running on GCE node
I20260927 13:59:30.456192 5897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:30.457103 5897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:30.458360 5897 hybrid_clock.cc:648] HybridClock initialized: now 1790517570458371 us; error 95 us; skew 500 ppm
I20260927 13:59:30.458704 5897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:30 dist-test-slave-gkks krb5kdc[5657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517570, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:30.461834 5897 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:59:30.463310 5897 webserver.cc:460] Webserver started at http://127.22.26.194:41081/ using document root <none> and password file <none>
I20260927 13:59:30.463953 5897 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:30.464098 5897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:30.464409 5897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:59:30.467295 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "a158484037994d5eb0ef02d9bbd87a58"
format_stamp: "Formatted at 2026-09-27 13:59:30 on dist-test-slave-gkks"
server_key: "8bb3008b081bbce8a1d3a997b2b2c89d"
server_key_iv: "17d254fbf9863145a3f32d0406ce9892"
server_key_version: "encryptionkey@0"
I20260927 13:59:30.468041 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "a158484037994d5eb0ef02d9bbd87a58"
format_stamp: "Formatted at 2026-09-27 13:59:30 on dist-test-slave-gkks"
server_key: "8bb3008b081bbce8a1d3a997b2b2c89d"
server_key_iv: "17d254fbf9863145a3f32d0406ce9892"
server_key_version: "encryptionkey@0"
I20260927 13:59:30.472920 5897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:59:30.476300 5912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:30.477494 5897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:59:30.477663 5897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "a158484037994d5eb0ef02d9bbd87a58"
format_stamp: "Formatted at 2026-09-27 13:59:30 on dist-test-slave-gkks"
server_key: "8bb3008b081bbce8a1d3a997b2b2c89d"
server_key_iv: "17d254fbf9863145a3f32d0406ce9892"
server_key_version: "encryptionkey@0"
I20260927 13:59:30.477849 5897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:30.499822 5897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:30.504293 5897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:30.504611 5897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:30.505577 5897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:30.506880 5897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:30.506983 5897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:30.507172 5897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:30.507247 5897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:30.540516 5897 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:42591
I20260927 13:59:30.540606 6024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:42591 every 8 connection(s)
I20260927 13:59:30.541900 5897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:59:30.548475 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 5897
I20260927 13:59:30.548659 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 13:59:30.549132 22635 external_mini_cluster.cc:1428] Setting key a1992aa1223196c28bf983bd9898e2b7
Sep 27 13:59:30 dist-test-slave-gkks krb5kdc[5657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517570, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:30.563020 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:30.544422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:53715 (local address 127.22.26.254:42401)
0927 13:59:30.544638 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:30.544645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:30.545811 (+ 1166us) server_negotiation.cc:407] Connection header received
0927 13:59:30.547677 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:30.547682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:30.547755 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:30.547846 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:30.549591 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:30.550452 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:30.552243 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:30.553037 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:30.553182 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:30.556592 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:30.556601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:30.556608 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:30.556651 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:30.559749 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:30.560692 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:30.560697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:30.560699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:30.560823 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:30.561303 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:30.561306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:30.561308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:30.561583 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:30.561701 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:30.562600 (+ 899us) server_negotiation.cc:299] Negotiation successful
0927 13:59:30.562780 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:59:30.564512 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:30.544932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:53715)
0927 13:59:30.545651 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:30.545690 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:30.547403 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:30.548056 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:30.548064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:30.548647 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:59:30.549419 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:30.549438 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:30.550641 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:30.550646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:30.552096 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:30.552109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:30.553263 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:30.553269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:30.553487 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:30.554191 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:59:30.554213 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:59:30.556354 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:59:30.559882 (+ 3528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:30.559894 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:30.559911 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:30.560554 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:30.560939 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:30.560946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:30.560953 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:30.561191 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:30.561706 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:59:30.561716 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:59:30.562159 (+ 443us) client_negotiation.cc:769] Sending connection context
0927 13:59:30.562467 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 13:59:30.563031 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":271,"threads_started":1}
I20260927 13:59:30.566039 6025 heartbeater.cc:347] Connected to a master server at 127.22.26.254:42401
I20260927 13:59:30.566344 6025 heartbeater.cc:464] Registering TS with master...
I20260927 13:59:30.567055 6025 heartbeater.cc:511] Master 127.22.26.254:42401 requested a full tablet report, sending...
I20260927 13:59:30.568608 5702 ts_manager.cc:194] Registered new tserver with Master: a158484037994d5eb0ef02d9bbd87a58 (127.22.26.194:42591)
I20260927 13:59:30.569864 5702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:53715
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 13:59:30.607383 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:42401
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:30.932837 6030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:30.933117 6030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:30.933297 6030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:30.940975 6030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:30.941089 6030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:30.941309 6030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:59:30.951267 6030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:30.952602 6030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:30.953917 6030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:30.967406 6035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:30.967492 6036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:30.967666 6038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:30.968446 6030 server_base.cc:1034] running on GCE node
I20260927 13:59:30.969043 6030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:30.969964 6030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:30.971185 6030 hybrid_clock.cc:648] HybridClock initialized: now 1790517570971138 us; error 55 us; skew 500 ppm
I20260927 13:59:30.971524 6030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:30 dist-test-slave-gkks krb5kdc[5657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517570, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:30.974565 6030 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:59:30.976156 6030 webserver.cc:460] Webserver started at http://127.22.26.195:42005/ using document root <none> and password file <none>
I20260927 13:59:30.976878 6030 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:30.977021 6030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:30.977319 6030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:59:30.980224 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b"
format_stamp: "Formatted at 2026-09-27 13:59:30 on dist-test-slave-gkks"
server_key: "a0c0db4fb5342bec71fef268caea9f15"
server_key_iv: "01202bb36e0d0f92413262245a0889f9"
server_key_version: "encryptionkey@0"
I20260927 13:59:30.981037 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b"
format_stamp: "Formatted at 2026-09-27 13:59:30 on dist-test-slave-gkks"
server_key: "a0c0db4fb5342bec71fef268caea9f15"
server_key_iv: "01202bb36e0d0f92413262245a0889f9"
server_key_version: "encryptionkey@0"
I20260927 13:59:30.986330 6030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 13:59:30.989794 6045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:30.991173 6030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:59:30.991340 6030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b"
format_stamp: "Formatted at 2026-09-27 13:59:30 on dist-test-slave-gkks"
server_key: "a0c0db4fb5342bec71fef268caea9f15"
server_key_iv: "01202bb36e0d0f92413262245a0889f9"
server_key_version: "encryptionkey@0"
I20260927 13:59:30.991541 6030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:31.015620 6030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:31.020171 6030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:31.020571 6030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:31.021641 6030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:31.023061 6030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:31.023196 6030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 13:59:31.023335 6030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:31.023416 6030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:31.057787 6030 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:39767
I20260927 13:59:31.057837 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:39767 every 8 connection(s)
I20260927 13:59:31.059150 6030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:59:31.060642 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 6030
I20260927 13:59:31.060828 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 13:59:31.061321 22635 external_mini_cluster.cc:1428] Setting key 8aeaf1659f1e01c65bd4d842e0c0b53f
I20260927 13:59:31.066509 5891 heartbeater.cc:503] Master 127.22.26.254:42401 was elected leader, sending a full tablet report...
Sep 27 13:59:31 dist-test-slave-gkks krb5kdc[5657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517570, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:31.079629 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.061941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:46485 (local address 127.22.26.254:42401)
0927 13:59:31.062164 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:31.062170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:31.063625 (+ 1455us) server_negotiation.cc:407] Connection header received
0927 13:59:31.065708 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:31.065712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:31.065778 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:31.065859 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:31.067525 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.068418 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.069875 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.070266 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.070361 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.073686 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:31.073691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:31.073695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:31.073726 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:31.076946 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:31.077618 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:31.077621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:31.077624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:31.077753 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:31.078116 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:31.078120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:31.078122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:31.078371 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:31.078510 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:31.079170 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 13:59:31.079337 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 13:59:31.080577 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.062486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:46485)
0927 13:59:31.063455 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:31.063501 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:31.065469 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:31.066020 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:31.066028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:31.066622 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:59:31.067340 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:31.067364 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.068555 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.068559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:31.069750 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:31.069761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.070477 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.070483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:31.070670 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.071363 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:59:31.071385 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:59:31.073487 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:59:31.077091 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:31.077098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:31.077110 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:31.077511 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:31.077842 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:31.077846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:31.077848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:31.078012 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:31.078484 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:59:31.078489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:59:31.078799 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 13:59:31.079010 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 13:59:31.079450 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":356,"threads_started":1}
I20260927 13:59:31.081943 6158 heartbeater.cc:347] Connected to a master server at 127.22.26.254:42401
I20260927 13:59:31.082237 6158 heartbeater.cc:464] Registering TS with master...
I20260927 13:59:31.082998 6158 heartbeater.cc:511] Master 127.22.26.254:42401 requested a full tablet report, sending...
I20260927 13:59:31.084442 5702 ts_manager.cc:194] Registered new tserver with Master: ab7c5f711c9b4dd4b1c4b8b376116d5b (127.22.26.195:39767)
I20260927 13:59:31.085317 5702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:46485
I20260927 13:59:31.089246 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:59:31.108336 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.093898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.22.26.254:42401)
0927 13:59:31.094065 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:31.094070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:31.094558 (+ 488us) server_negotiation.cc:407] Connection header received
0927 13:59:31.094693 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:31.094697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:31.094754 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:31.094834 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:31.096415 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.097517 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.099835 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.100366 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.100476 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.102935 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:31.102940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:31.102943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:31.102979 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:31.105746 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:31.106528 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:31.106532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:31.106535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:31.106666 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:31.106996 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:31.106999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:31.107001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:31.107428 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:31.107668 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:31.107907 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 13:59:31.108077 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:59:31.112342 5702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:59:31.119262 5702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:59:31.153640 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.132736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.22.26.195:39767)
0927 13:59:31.143936 (+ 11200us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:31.143949 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:31.143992 (+ 43us) server_negotiation.cc:407] Connection header received
0927 13:59:31.144125 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:31.144136 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:31.144483 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:31.145619 (+ 1136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:31.145967 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.147841 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.151139 (+ 3298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.152403 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.152540 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.152783 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:31.153149 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 13:59:31.153321 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10973,"thread_start_us":10718,"threads_started":1}
I20260927 13:59:31.153834 6170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.131974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:39767 (local address 127.0.0.1:58106)
0927 13:59:31.142909 (+ 10935us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:31.142932 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:31.143116 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:31.144775 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:31.144787 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:31.144829 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:31.145365 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:31.145381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.148014 (+ 2633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.148019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:31.150620 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:31.150635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.152622 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.152629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:31.152873 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.152945 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 13:59:31.153093 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 13:59:31.153300 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10788,"spinlock_wait_cycles":15616,"thread_start_us":9643,"threads_started":1}
I20260927 13:59:31.155478 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.131775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:33815 (local address 127.0.0.1:45344)
0927 13:59:31.142055 (+ 10280us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:31.142103 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:31.143119 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:31.146860 (+ 3741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:31.146869 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:31.146906 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:31.147462 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:31.147476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.150610 (+ 3134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.150617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:31.152439 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:31.152452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.154563 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.154571 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:31.154749 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.154786 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 13:59:31.154858 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 13:59:31.154982 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9973,"spinlock_wait_cycles":2944,"thread_start_us":10856,"threads_started":1}
I20260927 13:59:31.156418 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.144309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:42591 (local address 127.0.0.1:34822)
0927 13:59:31.148109 (+ 3800us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:31.148125 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:31.148303 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:31.149205 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:31.149210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:31.149241 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:31.149734 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:31.149749 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.151730 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.151736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:31.153786 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:31.153804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.155582 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.155591 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:31.155802 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.155847 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 13:59:31.155959 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 13:59:31.156086 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3651,"thread_start_us":3510,"threads_started":1}
I20260927 13:59:31.156510 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.143284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.22.26.194:42591)
0927 13:59:31.147622 (+ 4338us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:31.147632 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:31.148256 (+ 624us) server_negotiation.cc:407] Connection header received
0927 13:59:31.148505 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:31.148511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:31.148696 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:31.148925 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:31.149967 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.151553 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.153967 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.155346 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.155494 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.155688 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:31.156014 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 13:59:31.156188 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4152,"thread_start_us":4020,"threads_started":1}
I20260927 13:59:31.156981 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.132102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.22.26.193:33815)
0927 13:59:31.144296 (+ 12194us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:31.144305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:31.144336 (+ 31us) server_negotiation.cc:407] Connection header received
0927 13:59:31.144462 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:31.144474 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:31.146256 (+ 1782us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:31.146523 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:31.147671 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.150441 (+ 2770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.152704 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.154295 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.155897 (+ 1602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.156170 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:31.156247 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 13:59:31.156430 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11999,"thread_start_us":11837,"threads_started":1}
I20260927 13:59:31.158172 6093 tablet_service.cc:1467] Processing CreateTablet for tablet 8c772b71edb34036a0d372c0fef4a049 (DEFAULT_TABLE table=test-table [id=c5b2240779d2480bb4405c78a63b53ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:59:31.159788 6093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c772b71edb34036a0d372c0fef4a049. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:31.160216 5960 tablet_service.cc:1467] Processing CreateTablet for tablet 8c772b71edb34036a0d372c0fef4a049 (DEFAULT_TABLE table=test-table [id=c5b2240779d2480bb4405c78a63b53ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:59:31.160308 5826 tablet_service.cc:1467] Processing CreateTablet for tablet 8c772b71edb34036a0d372c0fef4a049 (DEFAULT_TABLE table=test-table [id=c5b2240779d2480bb4405c78a63b53ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:59:31.161535 5960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c772b71edb34036a0d372c0fef4a049. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:31.161772 5826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c772b71edb34036a0d372c0fef4a049. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:31.172014 6176 tablet_bootstrap.cc:492] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58: Bootstrap starting.
I20260927 13:59:31.175024 6176 tablet_bootstrap.cc:654] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:31.175657 6177 tablet_bootstrap.cc:492] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Bootstrap starting.
I20260927 13:59:31.176622 6178 tablet_bootstrap.cc:492] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb: Bootstrap starting.
I20260927 13:59:31.177908 6176 log.cc:824] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:31.179286 6177 tablet_bootstrap.cc:654] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:31.179967 6178 tablet_bootstrap.cc:654] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:31.180473 6177 log.cc:824] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:31.181201 6176 tablet_bootstrap.cc:492] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58: No bootstrap required, opened a new log
I20260927 13:59:31.181146 6178 log.cc:824] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:31.181552 6176 ts_tablet_manager.cc:1397] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260927 13:59:31.183925 6177 tablet_bootstrap.cc:492] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b: No bootstrap required, opened a new log
I20260927 13:59:31.184175 6177 ts_tablet_manager.cc:1397] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260927 13:59:31.185209 6178 tablet_bootstrap.cc:492] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb: No bootstrap required, opened a new log
I20260927 13:59:31.185482 6178 ts_tablet_manager.cc:1397] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 13:59:31.192518 6176 raft_consensus.cc:348] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:31.192973 6176 raft_consensus.cc:374] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:31.193120 6176 raft_consensus.cc:729] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a158484037994d5eb0ef02d9bbd87a58, State: Initialized, Role: FOLLOWER
I20260927 13:59:31.193609 6178 raft_consensus.cc:348] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:31.193744 6176 consensus_queue.cc:260] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:31.193951 6178 raft_consensus.cc:374] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:31.194079 6178 raft_consensus.cc:729] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3d36d518f14027ba4f7c458fe7b8fb, State: Initialized, Role: FOLLOWER
I20260927 13:59:31.194423 6177 raft_consensus.cc:348] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:31.194890 6177 raft_consensus.cc:374] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:31.194765 6178 consensus_queue.cc:260] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:31.194983 6025 heartbeater.cc:503] Master 127.22.26.254:42401 was elected leader, sending a full tablet report...
I20260927 13:59:31.195020 6177 raft_consensus.cc:729] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7c5f711c9b4dd4b1c4b8b376116d5b, State: Initialized, Role: FOLLOWER
I20260927 13:59:31.195886 6176 ts_tablet_manager.cc:1428] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 13:59:31.195789 6177 consensus_queue.cc:260] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:31.197458 6158 heartbeater.cc:503] Master 127.22.26.254:42401 was elected leader, sending a full tablet report...
I20260927 13:59:31.197587 6178 ts_tablet_manager.cc:1428] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 13:59:31.198745 6177 ts_tablet_manager.cc:1428] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260927 13:59:31.224000 6182 raft_consensus.cc:482] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:59:31.224301 6182 raft_consensus.cc:504] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:31.225857 6182 leader_election.cc:288] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b3d36d518f14027ba4f7c458fe7b8fb (127.22.26.193:33815), ab7c5f711c9b4dd4b1c4b8b376116d5b (127.22.26.195:39767)
I20260927 13:59:31.233032 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.226342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:52407 (local address 127.22.26.195:39767)
0927 13:59:31.226610 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:31.226620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:31.227536 (+ 916us) server_negotiation.cc:407] Connection header received
0927 13:59:31.227739 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:31.227746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:31.227808 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:31.227901 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:31.228534 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.229388 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.231367 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.232112 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.232286 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.232369 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:31.232644 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 13:59:31.232764 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260927 13:59:31.233197 6186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.226578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:39767 (local address 127.22.26.194:52407)
0927 13:59:31.227438 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:31.227462 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:31.227607 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:31.227913 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:31.227918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:31.227940 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:31.228375 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:31.228385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.229517 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.229524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:31.231208 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:31.231220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.232334 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.232340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:31.232482 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.232505 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 13:59:31.232621 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 13:59:31.232756 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":50,"thread_start_us":490,"threads_started":1}
I20260927 13:59:31.233386 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.226533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:53655 (local address 127.22.26.193:33815)
0927 13:59:31.226748 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:31.226756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:31.227452 (+ 696us) server_negotiation.cc:407] Connection header received
0927 13:59:31.227681 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:31.227686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:31.227750 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:31.227856 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:31.228516 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.229399 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.231555 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.232388 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.232585 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.232673 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:31.232982 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 13:59:31.233119 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 13:59:31.233554 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.226365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:33815 (local address 127.22.26.194:53655)
0927 13:59:31.227350 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:31.227368 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:31.227507 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:31.227871 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:31.227876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:31.227903 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:31.228336 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:31.228347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.229531 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.229539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:31.231381 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:31.231395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.232646 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.232652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:31.232809 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.232834 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 13:59:31.232955 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 13:59:31.233094 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":3,"thread_start_us":530,"threads_started":1}
I20260927 13:59:31.234273 6113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c772b71edb34036a0d372c0fef4a049" candidate_uuid: "a158484037994d5eb0ef02d9bbd87a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" is_pre_election: true
I20260927 13:59:31.234699 5846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c772b71edb34036a0d372c0fef4a049" candidate_uuid: "a158484037994d5eb0ef02d9bbd87a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" is_pre_election: true
I20260927 13:59:31.234932 6113 raft_consensus.cc:2454] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a158484037994d5eb0ef02d9bbd87a58 in term 0.
I20260927 13:59:31.235368 5846 raft_consensus.cc:2454] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a158484037994d5eb0ef02d9bbd87a58 in term 0.
I20260927 13:59:31.235697 5914 leader_election.cc:302] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a158484037994d5eb0ef02d9bbd87a58, ab7c5f711c9b4dd4b1c4b8b376116d5b; no voters:
I20260927 13:59:31.236150 6182 raft_consensus.cc:2781] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:59:31.236342 6182 raft_consensus.cc:482] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:59:31.236480 6182 raft_consensus.cc:3037] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:31.238974 6182 raft_consensus.cc:504] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:31.239622 6182 leader_election.cc:288] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [CANDIDATE]: Term 1 election: Requested vote from peers 3b3d36d518f14027ba4f7c458fe7b8fb (127.22.26.193:33815), ab7c5f711c9b4dd4b1c4b8b376116d5b (127.22.26.195:39767)
I20260927 13:59:31.240139 6113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c772b71edb34036a0d372c0fef4a049" candidate_uuid: "a158484037994d5eb0ef02d9bbd87a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b"
I20260927 13:59:31.240135 5846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c772b71edb34036a0d372c0fef4a049" candidate_uuid: "a158484037994d5eb0ef02d9bbd87a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb"
I20260927 13:59:31.240336 6113 raft_consensus.cc:3037] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:31.240411 5846 raft_consensus.cc:3037] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:31.242944 6113 raft_consensus.cc:2454] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a158484037994d5eb0ef02d9bbd87a58 in term 1.
I20260927 13:59:31.243348 5914 leader_election.cc:302] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a158484037994d5eb0ef02d9bbd87a58, ab7c5f711c9b4dd4b1c4b8b376116d5b; no voters:
I20260927 13:59:31.243376 5846 raft_consensus.cc:2454] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a158484037994d5eb0ef02d9bbd87a58 in term 1.
I20260927 13:59:31.243642 6182 raft_consensus.cc:2781] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:59:31.244117 6182 raft_consensus.cc:686] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [term 1 LEADER]: Becoming Leader. State: Replica: a158484037994d5eb0ef02d9bbd87a58, State: Running, Role: LEADER
I20260927 13:59:31.244624 6182 consensus_queue.cc:237] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:31.249359 5702 catalog_manager.cc:5495] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to a158484037994d5eb0ef02d9bbd87a58 (127.22.26.194). New cstate: current_term: 1 leader_uuid: "a158484037994d5eb0ef02d9bbd87a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } health_report { overall_health: HEALTHY } } }
I20260927 13:59:31.273887 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.268259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34836 (local address 127.22.26.194:42591)
0927 13:59:31.268478 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:31.268484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:31.268566 (+ 82us) server_negotiation.cc:407] Connection header received
0927 13:59:31.268828 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:31.268832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:31.268917 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:31.269071 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:59:31.269452 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.270159 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.271702 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.272120 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.272261 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.272770 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:59:31.273287 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:59:31.273451 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:31.273520 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 13:59:31.273603 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:59:31.282451 6113 raft_consensus.cc:1261] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 1 FOLLOWER]: Refusing update from remote peer a158484037994d5eb0ef02d9bbd87a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:59:31.282512 5846 raft_consensus.cc:1261] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 1 FOLLOWER]: Refusing update from remote peer a158484037994d5eb0ef02d9bbd87a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:59:31.283541 6182 consensus_queue.cc:1035] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:59:31.283838 6187 consensus_queue.cc:1035] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 13:59:31.284891 5892 tablet.cc:2369] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:59:31.293946 6026 tablet.cc:2369] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:59:31.295960 6190 mvcc.cc:205] Tried to move back new op lower bound from 7333959971962105856 to 7333959971822215168. Current Snapshot: MvccSnapshot[applied={T|T < 7333959971962105856}]
I20260927 13:59:31.310732 5701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:59:31.311147 5701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:59:31.315483 5701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Sending DeleteTablet for 3 replicas of tablet 8c772b71edb34036a0d372c0fef4a049
I20260927 13:59:31.316769 6093 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c772b71edb34036a0d372c0fef4a049 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:59:31 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:58106
I20260927 13:59:31.316867 5826 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c772b71edb34036a0d372c0fef4a049 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:59:31 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:45344
I20260927 13:59:31.317130 5960 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c772b71edb34036a0d372c0fef4a049 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:59:31 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:34822
I20260927 13:59:31.317395 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 5763
I20260927 13:59:31.329576 6200 tablet_replica.cc:330] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb: stopping tablet replica
I20260927 13:59:31.329955 6202 tablet_replica.cc:330] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58: stopping tablet replica
I20260927 13:59:31.330013 6201 tablet_replica.cc:330] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b: stopping tablet replica
I20260927 13:59:31.330255 6200 raft_consensus.cc:2229] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:59:31.330412 6202 raft_consensus.cc:2229] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:59:31.330444 6201 raft_consensus.cc:2229] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:59:31.330977 6200 raft_consensus.cc:2258] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:59:31.331018 6201 raft_consensus.cc:2258] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:59:31.331985 6202 raft_consensus.cc:2258] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:59:31.333447 6200 ts_tablet_manager.cc:1905] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:59:31.333453 6201 ts_tablet_manager.cc:1905] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 13:59:31 dist-test-slave-gkks krb5kdc[5657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 13:59:31.340834 6200 ts_tablet_manager.cc:1918] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:59:31.341171 6200 log.cc:1196] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/8c772b71edb34036a0d372c0fef4a049
I20260927 13:59:31.341787 6200 ts_tablet_manager.cc:1939] T 8c772b71edb34036a0d372c0fef4a049 P 3b3d36d518f14027ba4f7c458fe7b8fb: Deleting consensus metadata
I20260927 13:59:31.342797 6202 ts_tablet_manager.cc:1905] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:59:31.343896 5689 catalog_manager.cc:4841] TS 3b3d36d518f14027ba4f7c458fe7b8fb (127.22.26.193:33815): tablet 8c772b71edb34036a0d372c0fef4a049 (table test-table [id=c5b2240779d2480bb4405c78a63b53ad]) successfully deleted
I20260927 13:59:31.347038 6201 ts_tablet_manager.cc:1918] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:59:31.347302 6201 log.cc:1196] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/8c772b71edb34036a0d372c0fef4a049
I20260927 13:59:31.347949 6201 ts_tablet_manager.cc:1939] T 8c772b71edb34036a0d372c0fef4a049 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Deleting consensus metadata
I20260927 13:59:31.349135 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.318869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.22.26.193:33815)
0927 13:59:31.319126 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:31.319136 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:31.329479 (+ 10343us) server_negotiation.cc:407] Connection header received
0927 13:59:31.329726 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:31.329732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:31.329820 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:31.329893 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:31.332463 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.333536 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.335569 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.336208 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.336319 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.342250 (+ 5931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:31.342263 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:31.342279 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:31.342328 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:31.345630 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:31.346380 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:31.346389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:31.346392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:31.346537 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:31.347038 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:31.347043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:31.347046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:31.347565 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:31.347863 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:31.348542 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 13:59:31.348761 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:59:31.349867 5687 catalog_manager.cc:4841] TS ab7c5f711c9b4dd4b1c4b8b376116d5b (127.22.26.195:39767): tablet 8c772b71edb34036a0d372c0fef4a049 (table test-table [id=c5b2240779d2480bb4405c78a63b53ad]) successfully deleted
I20260927 13:59:31.350778 5886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:31.351042 6202 ts_tablet_manager.cc:1918] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:59:31.351361 6202 log.cc:1196] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/8c772b71edb34036a0d372c0fef4a049
I20260927 13:59:31.352173 6202 ts_tablet_manager.cc:1939] T 8c772b71edb34036a0d372c0fef4a049 P a158484037994d5eb0ef02d9bbd87a58: Deleting consensus metadata
I20260927 13:59:31.354983 5689 catalog_manager.cc:4841] TS a158484037994d5eb0ef02d9bbd87a58 (127.22.26.194:42591): tablet 8c772b71edb34036a0d372c0fef4a049 (table test-table [id=c5b2240779d2480bb4405c78a63b53ad]) successfully deleted
I20260927 13:59:31.460361 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 5763
I20260927 13:59:31.484308 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 5897
Sep 27 13:59:31 dist-test-slave-gkks krb5kdc[5657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 13:59:31.501470 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.485101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.22.26.194:42591)
0927 13:59:31.485287 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:31.485293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:31.485321 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:59:31.485467 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:31.485474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:31.485559 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:31.485659 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:31.486894 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.488227 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.489716 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.490447 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.490564 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.495573 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:31.495591 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:31.495607 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:31.495642 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:31.498725 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:31.499612 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:31.499618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:31.499621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:31.499754 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:31.500146 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:31.500149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:31.500151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:31.500474 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:31.500736 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:31.501018 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 13:59:31.501173 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:59:31.502350 6020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:31.618634 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 5897
I20260927 13:59:31.638303 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 6030
Sep 27 13:59:31 dist-test-slave-gkks krb5kdc[5657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 13:59:31.655390 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:31.639030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.22.26.195:39767)
0927 13:59:31.639338 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:31.639345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:31.639630 (+ 285us) server_negotiation.cc:407] Connection header received
0927 13:59:31.639865 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:31.639872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:31.639976 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:31.640111 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:31.641748 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.642537 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.644049 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:31.644469 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:31.644627 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:31.649500 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:31.649516 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:31.649534 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:31.649572 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:31.652513 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:31.653255 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:31.653263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:31.653265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:31.653400 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:31.653807 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:31.653812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:31.653814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:31.654188 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:31.654513 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:31.654875 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 13:59:31.655055 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 13:59:31.656116 6153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:31.745873 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 6030
I20260927 13:59:31.762691 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 5670
I20260927 13:59:31.763829 5732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:31.876609 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 5670
I20260927 13:59:51.898412 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42401
--webserver_interface=127.22.26.254
--webserver_port=46199
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:42401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:59:52.230871 6208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:52.231220 6208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:52.231386 6208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:52.238319 6208 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 13:59:52.238445 6208 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 13:59:52.238511 6208 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 13:59:52.238610 6208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:59:52.238713 6208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:52.238816 6208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:52.238888 6208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:59:52.238978 6208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:59:52.248731 6208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:42401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=46199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:52.250000 6208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:52.251339 6208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:52.274376 6217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:52.274467 6215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:52.274410 6214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:52.275183 6208 server_base.cc:1034] running on GCE node
I20260927 13:59:52.276132 6208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:52.277830 6208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:52.279167 6208 hybrid_clock.cc:648] HybridClock initialized: now 1790517592279023 us; error 47 us; skew 500 ppm
I20260927 13:59:52.279588 6208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:52 dist-test-slave-gkks krb5kdc[5657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517592, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:52.288038 6208 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 13:59:52.289893 6208 webserver.cc:460] Webserver started at http://127.22.26.254:46199/ using document root <none> and password file <none>
I20260927 13:59:52.290805 6208 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:52.290966 6208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:52.297526 6208 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 13:59:52.300819 6224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:52.302277 6208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:59:52.302485 6208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "d803bc2147c048cf8be3d45fd18fb911"
format_stamp: "Formatted at 2026-09-27 13:59:29 on dist-test-slave-gkks"
server_key: "4efdc8b9c3f39a9fdde08ab6e7ed1217"
server_key_iv: "3579a200d8c4b6115c3af4b46a0026fe"
server_key_version: "encryptionkey@0"
I20260927 13:59:52.303691 6208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:52.328437 6208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:52.332759 6208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:52.333086 6208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:52.352967 6208 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:42401
I20260927 13:59:52.352990 6275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:42401 every 8 connection(s)
I20260927 13:59:52.354619 6208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 13:59:52.361539 6276 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:59:52.362013 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 6208
I20260927 13:59:52.363020 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:33815
--local_ip_for_outbound_sockets=127.22.26.193
--tserver_master_addrs=127.22.26.254:42401
--webserver_port=38635
--webserver_interface=127.22.26.193
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:59:52.366557 6276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Bootstrap starting.
I20260927 13:59:52.387790 6276 log.cc:824] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:52.399785 6276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:59:52.400313 6276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Bootstrap complete.
I20260927 13:59:52.409688 6276 raft_consensus.cc:348] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } }
I20260927 13:59:52.410521 6276 raft_consensus.cc:729] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d803bc2147c048cf8be3d45fd18fb911, State: Initialized, Role: FOLLOWER
I20260927 13:59:52.411250 6276 consensus_queue.cc:260] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } }
I20260927 13:59:52.411500 6276 raft_consensus.cc:388] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:59:52.411650 6276 raft_consensus.cc:482] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:59:52.411849 6276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:59:52.415999 6276 raft_consensus.cc:504] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } }
I20260927 13:59:52.416556 6276 leader_election.cc:302] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d803bc2147c048cf8be3d45fd18fb911; no voters:
I20260927 13:59:52.417196 6276 leader_election.cc:288] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 13:59:52.417346 6280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:59:52.418565 6280 raft_consensus.cc:686] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 2 LEADER]: Becoming Leader. State: Replica: d803bc2147c048cf8be3d45fd18fb911, State: Running, Role: LEADER
I20260927 13:59:52.419214 6280 consensus_queue.cc:237] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } }
I20260927 13:59:52.419591 6276 sys_catalog.cc:564] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:59:52.421685 6282 sys_catalog.cc:455] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d803bc2147c048cf8be3d45fd18fb911. Latest consensus state: current_term: 2 leader_uuid: "d803bc2147c048cf8be3d45fd18fb911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } } }
I20260927 13:59:52.422076 6282 sys_catalog.cc:458] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: This master's current role is: LEADER
I20260927 13:59:52.422668 6281 sys_catalog.cc:455] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d803bc2147c048cf8be3d45fd18fb911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } } }
I20260927 13:59:52.422982 6281 sys_catalog.cc:458] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: This master's current role is: LEADER
I20260927 13:59:52.427903 6286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:59:52.437264 6286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:59:52.438796 6286 catalog_manager.cc:1259] Loaded cluster ID: 4cab217f3b4e43bcb9b65e13c688fd44
I20260927 13:59:52.438971 6286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:59:52.443137 6286 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:59:52.444739 6286 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Loaded TSK: 0
I20260927 13:59:52.474074 6286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Generated new TSK 1
I20260927 13:59:52.474447 6286 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 13:59:52.728102 6278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:52.728538 6278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:52.728757 6278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:52.737998 6278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:52.738147 6278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:52.738348 6278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 13:59:52.748466 6278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:33815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=38635
--tserver_master_addrs=127.22.26.254:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:52.749788 6278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:52.751170 6278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:52.767452 6304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:52.767681 6303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:52.768224 6306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:52.768585 6278 server_base.cc:1034] running on GCE node
I20260927 13:59:52.769292 6278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:52.770334 6278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:52.771551 6278 hybrid_clock.cc:648] HybridClock initialized: now 1790517592771485 us; error 53 us; skew 500 ppm
I20260927 13:59:52.771876 6278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:52 dist-test-slave-gkks krb5kdc[5657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517592, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:52.775154 6278 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 13:59:52.776782 6278 webserver.cc:460] Webserver started at http://127.22.26.193:38635/ using document root <none> and password file <none>
I20260927 13:59:52.777496 6278 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:52.777638 6278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:52.783190 6278 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:59:52.786077 6313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:52.787582 6278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 13:59:52.787755 6278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "3b3d36d518f14027ba4f7c458fe7b8fb"
format_stamp: "Formatted at 2026-09-27 13:59:29 on dist-test-slave-gkks"
server_key: "93efd19c5d88f27f12f4116f9c11f58c"
server_key_iv: "35ab40ae05d64831c3d14d7c53f45cd5"
server_key_version: "encryptionkey@0"
I20260927 13:59:52.788722 6278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:52.806695 6278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:52.811257 6278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:52.811524 6278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:52.812592 6278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:52.814044 6278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:52.814168 6278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:52.814299 6278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:52.814394 6278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:52.854862 6278 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:33815
I20260927 13:59:52.854939 6425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:33815 every 8 connection(s)
I20260927 13:59:52.856405 6278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 13:59:52.860252 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 6278
I20260927 13:59:52.861090 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:42591
--local_ip_for_outbound_sockets=127.22.26.194
--tserver_master_addrs=127.22.26.254:42401
--webserver_port=41081
--webserver_interface=127.22.26.194
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:59:52 dist-test-slave-gkks krb5kdc[5657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517592, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:52.883940 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:52.860078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:57597)
0927 13:59:52.861014 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:52.861066 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:52.863847 (+ 2781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:52.865373 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:52.865386 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:52.866216 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:59:52.867030 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:52.867060 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:52.868942 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:52.868951 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:52.870297 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:52.870310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:52.871874 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:52.871879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:52.872083 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:52.872776 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:59:52.872798 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:59:52.874924 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:59:52.879674 (+ 4750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:52.879681 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:52.879695 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:52.880122 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:52.880661 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:52.880664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:52.880666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:52.880838 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:52.881866 (+ 1028us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:59:52.881872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:59:52.882178 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 13:59:52.882406 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 13:59:52.882797 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":335,"threads_started":1}
I20260927 13:59:52.884554 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:52.859636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:57597 (local address 127.22.26.254:42401)
0927 13:59:52.860353 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:52.860372 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:52.861220 (+ 848us) server_negotiation.cc:407] Connection header received
0927 13:59:52.864465 (+ 3245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:52.864480 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:52.864843 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:52.865155 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:52.867296 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:52.868769 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:52.870449 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:52.871666 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:52.871859 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:52.875241 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:52.875253 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:52.875274 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:52.875324 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:52.879496 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:52.880273 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:52.880285 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:52.880297 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:52.880532 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:52.880952 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:52.880960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:52.880966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:52.881709 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:52.881931 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:52.882688 (+ 757us) server_negotiation.cc:299] Negotiation successful
0927 13:59:52.883192 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":307,"threads_started":1}
I20260927 13:59:52.887648 6426 heartbeater.cc:347] Connected to a master server at 127.22.26.254:42401
I20260927 13:59:52.888008 6426 heartbeater.cc:464] Registering TS with master...
I20260927 13:59:52.889029 6426 heartbeater.cc:511] Master 127.22.26.254:42401 requested a full tablet report, sending...
I20260927 13:59:52.891593 6241 ts_manager.cc:194] Registered new tserver with Master: 3b3d36d518f14027ba4f7c458fe7b8fb (127.22.26.193:33815)
I20260927 13:59:52.893596 6241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:57597
W20260927 13:59:53.194615 6430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:53.194880 6430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:53.194972 6430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:53.202127 6430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:53.202226 6430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:53.202337 6430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 13:59:53.211956 6430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:42591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=41081
--tserver_master_addrs=127.22.26.254:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:53.213171 6430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:53.214455 6430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:53.227638 6436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:53.227697 6438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:53.228051 6435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:53.228422 6430 server_base.cc:1034] running on GCE node
I20260927 13:59:53.229156 6430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:53.230201 6430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:53.231403 6430 hybrid_clock.cc:648] HybridClock initialized: now 1790517593231344 us; error 41 us; skew 500 ppm
I20260927 13:59:53.231742 6430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:53 dist-test-slave-gkks krb5kdc[5657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517593, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:53.234897 6430 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 13:59:53.236332 6430 webserver.cc:460] Webserver started at http://127.22.26.194:41081/ using document root <none> and password file <none>
I20260927 13:59:53.237069 6430 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:53.237187 6430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:53.242796 6430 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:59:53.245671 6445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:53.247008 6430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:59:53.247224 6430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "a158484037994d5eb0ef02d9bbd87a58"
format_stamp: "Formatted at 2026-09-27 13:59:30 on dist-test-slave-gkks"
server_key: "8bb3008b081bbce8a1d3a997b2b2c89d"
server_key_iv: "17d254fbf9863145a3f32d0406ce9892"
server_key_version: "encryptionkey@0"
I20260927 13:59:53.248243 6430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:53.265825 6430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:53.270406 6430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:53.270630 6430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:53.271605 6430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:53.272940 6430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:53.273025 6430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:53.273094 6430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:53.273160 6430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:53.306270 6430 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:42591
I20260927 13:59:53.306315 6557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:42591 every 8 connection(s)
I20260927 13:59:53.307942 6430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 13:59:53.309052 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 6430
I20260927 13:59:53.309692 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:39767
--local_ip_for_outbound_sockets=127.22.26.195
--tserver_master_addrs=127.22.26.254:42401
--webserver_port=42005
--webserver_interface=127.22.26.195
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:59:53 dist-test-slave-gkks krb5kdc[5657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517593, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:53.329654 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.310780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:53693 (local address 127.22.26.254:42401)
0927 13:59:53.310956 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:53.310961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:53.312575 (+ 1614us) server_negotiation.cc:407] Connection header received
0927 13:59:53.314472 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:53.314478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:53.314588 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:53.314705 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:53.316375 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.317094 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.318756 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.319254 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.319392 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.323589 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:53.323594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:53.323598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:53.323632 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:53.326348 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:53.327194 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:53.327199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:53.327201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:53.327330 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:53.327786 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:53.327795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:53.327797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:53.328126 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:53.328293 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:53.329036 (+ 743us) server_negotiation.cc:299] Negotiation successful
0927 13:59:53.329285 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:59:53.330655 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.311483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:53693)
0927 13:59:53.312394 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:53.312436 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:53.314176 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:53.314967 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:53.314975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:53.315600 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:59:53.316189 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.316207 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.317257 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.317270 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.318502 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.318515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.319496 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.319505 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.319778 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.320619 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:59:53.320644 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:59:53.323298 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:59:53.326516 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:53.326525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:53.326545 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:53.327019 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:53.327449 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:53.327454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:53.327458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:53.327658 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:53.328275 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:59:53.328283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:59:53.328661 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 13:59:53.328936 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 13:59:53.329518 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":356,"threads_started":1}
I20260927 13:59:53.332234 6558 heartbeater.cc:347] Connected to a master server at 127.22.26.254:42401
I20260927 13:59:53.332579 6558 heartbeater.cc:464] Registering TS with master...
I20260927 13:59:53.333303 6558 heartbeater.cc:511] Master 127.22.26.254:42401 requested a full tablet report, sending...
I20260927 13:59:53.334908 6241 ts_manager.cc:194] Registered new tserver with Master: a158484037994d5eb0ef02d9bbd87a58 (127.22.26.194:42591)
I20260927 13:59:53.336189 6241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:53693
W20260927 13:59:53.655342 6561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:59:53.655627 6561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:59:53.655730 6561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:59:53.662873 6561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:59:53.662986 6561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:59:53.663164 6561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 13:59:53.673085 6561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:39767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=42005
--tserver_master_addrs=127.22.26.254:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:59:53.674343 6561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:59:53.675724 6561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:59:53.688899 6566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:53.689054 6569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:59:53.689128 6567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:59:53.689572 6561 server_base.cc:1034] running on GCE node
I20260927 13:59:53.690235 6561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:59:53.691279 6561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:59:53.692469 6561 hybrid_clock.cc:648] HybridClock initialized: now 1790517593692407 us; error 55 us; skew 500 ppm
I20260927 13:59:53.692796 6561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:59:53 dist-test-slave-gkks krb5kdc[5657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517593, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:59:53.695948 6561 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 13:59:53.697463 6561 webserver.cc:460] Webserver started at http://127.22.26.195:42005/ using document root <none> and password file <none>
I20260927 13:59:53.698158 6561 fs_manager.cc:359] Metadata directory not provided
I20260927 13:59:53.698300 6561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:59:53.703704 6561 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:59:53.706430 6576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:53.707908 6561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:59:53.708081 6561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b"
format_stamp: "Formatted at 2026-09-27 13:59:30 on dist-test-slave-gkks"
server_key: "a0c0db4fb5342bec71fef268caea9f15"
server_key_iv: "01202bb36e0d0f92413262245a0889f9"
server_key_version: "encryptionkey@0"
I20260927 13:59:53.709116 6561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:59:53.722698 6561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:59:53.726559 6561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:59:53.726822 6561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:59:53.727828 6561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:59:53.729120 6561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:59:53.729255 6561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:53.729388 6561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:59:53.729478 6561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:59:53.763469 6561 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:39767
I20260927 13:59:53.763518 6688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:39767 every 8 connection(s)
I20260927 13:59:53.765026 6561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 13:59:53.766495 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 6561
Sep 27 13:59:53 dist-test-slave-gkks krb5kdc[5657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517593, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 13:59:53.789800 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.767594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.22.26.254:42401)
0927 13:59:53.768068 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:53.768078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:53.774793 (+ 6715us) server_negotiation.cc:407] Connection header received
0927 13:59:53.774974 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:53.774978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:53.775025 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:53.775172 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:53.777222 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.778157 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.779587 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.780088 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.780224 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.783624 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:53.783629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:53.783632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:53.783666 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:53.786913 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:53.787654 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:53.787664 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:53.787666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:53.787796 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:53.788246 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:53.788250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:53.788252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:53.788541 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:53.788828 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:53.789153 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 13:59:53.789378 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":307}
I20260927 13:59:53.790195 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.767718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:37573 (local address 127.22.26.254:42401)
0927 13:59:53.771578 (+ 3860us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:53.771587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:53.772359 (+ 772us) server_negotiation.cc:407] Connection header received
0927 13:59:53.774264 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:53.774274 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:53.774382 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:53.774513 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:53.776804 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.777735 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.779437 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.779941 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.780084 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.783912 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:53.783917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:53.783920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:53.783960 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:53.786983 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:53.787808 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:53.787812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:53.787815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:53.787924 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:53.788383 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:53.788387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:53.788392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:53.788649 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:53.788805 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:53.789515 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 13:59:53.789717 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":3702,"thread_start_us":3579,"threads_started":1}
I20260927 13:59:53.791971 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.768249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:37573)
0927 13:59:53.772169 (+ 3920us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:53.772218 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:53.773976 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:53.774819 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:53.774832 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:53.775699 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:59:53.776590 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.776622 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.777894 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.777900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.779287 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.779300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.780173 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.780182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.780441 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.781199 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:59:53.781222 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:59:53.783669 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:59:53.787172 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:53.787183 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:53.787202 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:53.787659 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:53.788045 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:59:53.788050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:59:53.788053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:59:53.788256 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:59:53.788778 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:59:53.788785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:59:53.789156 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 13:59:53.789420 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 13:59:53.790409 (+ 989us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3620,"thread_start_us":3433,"threads_started":1}
I20260927 13:59:53.793509 6689 heartbeater.cc:347] Connected to a master server at 127.22.26.254:42401
I20260927 13:59:53.793838 6689 heartbeater.cc:464] Registering TS with master...
I20260927 13:59:53.794559 6689 heartbeater.cc:511] Master 127.22.26.254:42401 requested a full tablet report, sending...
I20260927 13:59:53.796005 6241 ts_manager.cc:194] Registered new tserver with Master: ab7c5f711c9b4dd4b1c4b8b376116d5b (127.22.26.195:39767)
I20260927 13:59:53.796861 6241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:37573
I20260927 13:59:53.805471 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:59:53.813364 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.806630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45986 (local address 127.22.26.254:42401)
0927 13:59:53.806838 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:53.806842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:53.807557 (+ 715us) server_negotiation.cc:407] Connection header received
0927 13:59:53.807840 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:53.807844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:53.808021 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:53.808240 (+ 219us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:59:53.808723 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.809555 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.811369 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.811809 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.811913 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.812469 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:59:53.812693 (+ 224us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 13:59:53.812997 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45986: token expired
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 13:59:53.813546 6692 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45986: token expired
W20260927 13:59:53.813999 6163 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.26.254:42401)
I20260927 13:59:53.827682 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.814488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.22.26.254:42401)
0927 13:59:53.814698 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:53.814703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:53.814838 (+ 135us) server_negotiation.cc:407] Connection header received
0927 13:59:53.815020 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:53.815025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:53.815155 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:53.815251 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:53.816682 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.817392 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.818762 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.819267 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.819400 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.822219 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:53.822224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:53.822227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:53.822259 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:53.824999 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:53.825658 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:53.825665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:53.825669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:53.825798 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:53.826182 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:53.826188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:53.826191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:53.826504 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:53.826783 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:53.827124 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 13:59:53.827344 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 13:59:53.831552 6241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:59:53.838928 6241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:59:53.874995 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.851394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:39767 (local address 127.0.0.1:53308)
0927 13:59:53.862742 (+ 11348us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:53.862761 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:53.862870 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:53.864122 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:53.864133 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:53.864182 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:53.864710 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.864725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.867515 (+ 2790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.867524 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.870658 (+ 3134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.870672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.873573 (+ 2901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.873582 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.873879 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.873952 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 13:59:53.874115 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 13:59:53.874376 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11205,"mutex_wait_us":928,"thread_start_us":9199,"threads_started":1}
I20260927 13:59:53.876354 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.851950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.22.26.195:39767)
0927 13:59:53.852869 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:53.852882 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:53.862996 (+ 10114us) server_negotiation.cc:407] Connection header received
0927 13:59:53.863153 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:53.863164 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:53.863489 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:53.863732 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:53.865077 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.867252 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.870881 (+ 3629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.872902 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.873067 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.873372 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:53.875817 (+ 2445us) server_negotiation.cc:299] Negotiation successful
0927 13:59:53.876007 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":549,"threads_started":1}
I20260927 13:59:53.879987 6624 tablet_service.cc:1467] Processing CreateTablet for tablet 90b7294381da4370a2786b5af0206bb9 (DEFAULT_TABLE table=test-table [id=b350706e990240c7a48fc4acc44920df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:59:53.880649 6702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.851393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:33815 (local address 127.0.0.1:49398)
0927 13:59:53.861926 (+ 10533us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:53.861961 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:53.862133 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:53.871625 (+ 9492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:53.871633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:53.871670 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:53.872126 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.872141 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.875952 (+ 3811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.875957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.878069 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.878083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.879816 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.879823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.880008 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.880043 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:59:53.880156 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 13:59:53.880288 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10298,"thread_start_us":11055,"threads_started":1}
I20260927 13:59:53.880662 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.851407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.22.26.193:33815)
0927 13:59:53.870673 (+ 19266us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:53.870685 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:53.870720 (+ 35us) server_negotiation.cc:407] Connection header received
0927 13:59:53.870828 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:53.870839 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:53.871168 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:53.872368 (+ 1200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:53.872661 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.875762 (+ 3101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.878275 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.879560 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.879709 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.879954 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:53.880205 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 13:59:53.880349 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19052,"thread_start_us":19929,"threads_started":1}
I20260927 13:59:53.881567 6624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b7294381da4370a2786b5af0206bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:53.884135 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.863014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:42591 (local address 127.0.0.1:35142)
0927 13:59:53.876068 (+ 13054us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:53.876096 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:53.876231 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:53.876853 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:53.876858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:53.876887 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:53.877236 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.877247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.879245 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.879250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.881282 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.881297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.883332 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.883339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.883503 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.883534 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 13:59:53.883634 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 13:59:53.883755 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12914,"mutex_wait_us":1,"thread_start_us":12772,"threads_started":1}
I20260927 13:59:53.884186 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.863057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.22.26.194:42591)
0927 13:59:53.875478 (+ 12421us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:53.875486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:53.876226 (+ 740us) server_negotiation.cc:407] Connection header received
0927 13:59:53.876395 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:53.876404 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:53.876652 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:53.876857 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:53.877427 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.879005 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.881476 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.882890 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.883024 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.883231 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:53.883709 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 13:59:53.883861 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12250,"thread_start_us":12399,"threads_started":1}
I20260927 13:59:53.884805 6361 tablet_service.cc:1467] Processing CreateTablet for tablet 90b7294381da4370a2786b5af0206bb9 (DEFAULT_TABLE table=test-table [id=b350706e990240c7a48fc4acc44920df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:59:53.886076 6361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b7294381da4370a2786b5af0206bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:53.887887 6493 tablet_service.cc:1467] Processing CreateTablet for tablet 90b7294381da4370a2786b5af0206bb9 (DEFAULT_TABLE table=test-table [id=b350706e990240c7a48fc4acc44920df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:59:53.889204 6493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b7294381da4370a2786b5af0206bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:59:53.896898 6705 tablet_bootstrap.cc:492] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Bootstrap starting.
I20260927 13:59:53.897605 6426 heartbeater.cc:503] Master 127.22.26.254:42401 was elected leader, sending a full tablet report...
I20260927 13:59:53.902004 6705 tablet_bootstrap.cc:654] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:53.903604 6705 log.cc:824] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:53.903656 6707 tablet_bootstrap.cc:492] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58: Bootstrap starting.
I20260927 13:59:53.904603 6706 tablet_bootstrap.cc:492] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb: Bootstrap starting.
I20260927 13:59:53.907395 6707 tablet_bootstrap.cc:654] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:53.907807 6705 tablet_bootstrap.cc:492] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b: No bootstrap required, opened a new log
I20260927 13:59:53.908075 6706 tablet_bootstrap.cc:654] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb: Neither blocks nor log segments found. Creating new log.
I20260927 13:59:53.908107 6705 ts_tablet_manager.cc:1397] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260927 13:59:53.908717 6707 log.cc:824] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:53.909319 6706 log.cc:824] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb: Log is configured to *not* fsync() on all Append() calls
I20260927 13:59:53.911895 6707 tablet_bootstrap.cc:492] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58: No bootstrap required, opened a new log
I20260927 13:59:53.912348 6707 ts_tablet_manager.cc:1397] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260927 13:59:53.913805 6706 tablet_bootstrap.cc:492] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb: No bootstrap required, opened a new log
I20260927 13:59:53.914052 6706 ts_tablet_manager.cc:1397] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260927 13:59:53.916716 6705 raft_consensus.cc:348] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:53.917092 6705 raft_consensus.cc:374] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:53.917230 6705 raft_consensus.cc:729] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7c5f711c9b4dd4b1c4b8b376116d5b, State: Initialized, Role: FOLLOWER
I20260927 13:59:53.917852 6705 consensus_queue.cc:260] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:53.918956 6689 heartbeater.cc:503] Master 127.22.26.254:42401 was elected leader, sending a full tablet report...
I20260927 13:59:53.919610 6705 ts_tablet_manager.cc:1428] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260927 13:59:53.922457 6711 raft_consensus.cc:482] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:59:53.922796 6711 raft_consensus.cc:504] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:53.922766 6707 raft_consensus.cc:348] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:53.923395 6707 raft_consensus.cc:374] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:53.923501 6707 raft_consensus.cc:729] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a158484037994d5eb0ef02d9bbd87a58, State: Initialized, Role: FOLLOWER
I20260927 13:59:53.924306 6707 consensus_queue.cc:260] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:53.926509 6711 leader_election.cc:288] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b3d36d518f14027ba4f7c458fe7b8fb (127.22.26.193:33815), a158484037994d5eb0ef02d9bbd87a58 (127.22.26.194:42591)
I20260927 13:59:53.927805 6558 heartbeater.cc:503] Master 127.22.26.254:42401 was elected leader, sending a full tablet report...
I20260927 13:59:53.928601 6707 ts_tablet_manager.cc:1428] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58: Time spent starting tablet: real 0.016s user 0.012s sys 0.002s
I20260927 13:59:53.930305 6706 raft_consensus.cc:348] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:53.930848 6706 raft_consensus.cc:374] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:59:53.931020 6706 raft_consensus.cc:729] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3d36d518f14027ba4f7c458fe7b8fb, State: Initialized, Role: FOLLOWER
I20260927 13:59:53.931782 6706 consensus_queue.cc:260] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:53.933099 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.925127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:33815 (local address 127.22.26.195:35763)
0927 13:59:53.925415 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:53.925436 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:53.925557 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:53.926339 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:53.926343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:53.926366 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:53.926594 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.926609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.928523 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.928530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.930379 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.930392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.932372 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.932380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.932565 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.932602 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 13:59:53.932687 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 13:59:53.932822 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260927 13:59:53.933975 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.925193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:35763 (local address 127.22.26.193:33815)
0927 13:59:53.925798 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:53.925805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:53.925833 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:59:53.925911 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:53.925919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:53.926031 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:53.926137 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:53.926797 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.928358 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.930574 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.932052 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.932216 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.932285 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:53.933574 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0927 13:59:53.933714 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463}
I20260927 13:59:53.934715 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.925850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:59027 (local address 127.22.26.194:42591)
0927 13:59:53.926061 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:53.926068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:53.926830 (+ 762us) server_negotiation.cc:407] Connection header received
0927 13:59:53.927135 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:53.927140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:53.927237 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:53.927334 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:59:53.928180 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.929354 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.932417 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.933618 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.933723 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.933785 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:53.934329 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 13:59:53.934476 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 13:59:53.935397 6706 ts_tablet_manager.cc:1428] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb: Time spent starting tablet: real 0.021s user 0.014s sys 0.004s
I20260927 13:59:53.935424 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.925636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:42591 (local address 127.22.26.195:59027)
0927 13:59:53.926699 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0927 13:59:53.926721 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:59:53.926872 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:59:53.927507 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:59:53.927511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:59:53.927534 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:59:53.927913 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.927925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.930273 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.930280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.932259 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:59:53.932272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.933947 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.933954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:59:53.934134 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.934172 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 13:59:53.934925 (+ 753us) client_negotiation.cc:240] Negotiation successful
0927 13:59:53.935064 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":606,"threads_started":1}
I20260927 13:59:53.935196 6381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b7294381da4370a2786b5af0206bb9" candidate_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" is_pre_election: true
I20260927 13:59:53.935745 6381 raft_consensus.cc:2454] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7c5f711c9b4dd4b1c4b8b376116d5b in term 0.
I20260927 13:59:53.936342 6513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b7294381da4370a2786b5af0206bb9" candidate_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a158484037994d5eb0ef02d9bbd87a58" is_pre_election: true
I20260927 13:59:53.936465 6580 leader_election.cc:302] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b3d36d518f14027ba4f7c458fe7b8fb, ab7c5f711c9b4dd4b1c4b8b376116d5b; no voters:
I20260927 13:59:53.936829 6513 raft_consensus.cc:2454] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7c5f711c9b4dd4b1c4b8b376116d5b in term 0.
I20260927 13:59:53.936908 6711 raft_consensus.cc:2781] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:59:53.937037 6711 raft_consensus.cc:482] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:59:53.937155 6711 raft_consensus.cc:3037] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:53.939872 6711 raft_consensus.cc:504] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:53.940459 6711 leader_election.cc:288] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [CANDIDATE]: Term 1 election: Requested vote from peers 3b3d36d518f14027ba4f7c458fe7b8fb (127.22.26.193:33815), a158484037994d5eb0ef02d9bbd87a58 (127.22.26.194:42591)
I20260927 13:59:53.940945 6381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b7294381da4370a2786b5af0206bb9" candidate_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb"
I20260927 13:59:53.941015 6513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b7294381da4370a2786b5af0206bb9" candidate_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a158484037994d5eb0ef02d9bbd87a58"
I20260927 13:59:53.941167 6381 raft_consensus.cc:3037] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:53.941210 6513 raft_consensus.cc:3037] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:59:53.943408 6513 raft_consensus.cc:2454] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7c5f711c9b4dd4b1c4b8b376116d5b in term 1.
I20260927 13:59:53.943414 6381 raft_consensus.cc:2454] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7c5f711c9b4dd4b1c4b8b376116d5b in term 1.
I20260927 13:59:53.943813 6580 leader_election.cc:302] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b3d36d518f14027ba4f7c458fe7b8fb, ab7c5f711c9b4dd4b1c4b8b376116d5b; no voters:
I20260927 13:59:53.944118 6711 raft_consensus.cc:2781] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:59:53.944586 6711 raft_consensus.cc:686] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 1 LEADER]: Becoming Leader. State: Replica: ab7c5f711c9b4dd4b1c4b8b376116d5b, State: Running, Role: LEADER
I20260927 13:59:53.945070 6711 consensus_queue.cc:237] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } }
I20260927 13:59:53.949739 6240 catalog_manager.cc:5495] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b reported cstate change: term changed from 0 to 1, leader changed from <none> to ab7c5f711c9b4dd4b1c4b8b376116d5b (127.22.26.195). New cstate: current_term: 1 leader_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 } health_report { overall_health: UNKNOWN } } }
I20260927 13:59:53.987473 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:53.982236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.22.26.195:39767)
0927 13:59:53.982408 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:53.982413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:53.982595 (+ 182us) server_negotiation.cc:407] Connection header received
0927 13:59:53.982769 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:53.982773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:53.982850 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:53.982972 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:59:53.983370 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.984083 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.985498 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:53.985885 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:53.985976 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:53.986436 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:59:53.986952 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:59:53.987059 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:53.987211 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 13:59:53.987277 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:59:53.996379 6513 raft_consensus.cc:1261] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58 [term 1 FOLLOWER]: Refusing update from remote peer ab7c5f711c9b4dd4b1c4b8b376116d5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:59:53.996379 6381 raft_consensus.cc:1261] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 1 FOLLOWER]: Refusing update from remote peer ab7c5f711c9b4dd4b1c4b8b376116d5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:59:53.997432 6711 consensus_queue.cc:1035] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a158484037994d5eb0ef02d9bbd87a58" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 42591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:59:53.997807 6715 consensus_queue.cc:1035] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b3d36d518f14027ba4f7c458fe7b8fb" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:59:54.023695 6241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:59:54.023999 6241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:59:54.027422 6241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Sending DeleteTablet for 3 replicas of tablet 90b7294381da4370a2786b5af0206bb9
I20260927 13:59:54.028692 6624 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b7294381da4370a2786b5af0206bb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:59:54 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:53308
I20260927 13:59:54.028774 6361 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b7294381da4370a2786b5af0206bb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:59:54 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:49398
I20260927 13:59:54.028941 6493 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b7294381da4370a2786b5af0206bb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:59:54 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:35142
I20260927 13:59:54.029701 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 6278
I20260927 13:59:54.039740 6729 tablet_replica.cc:330] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb: stopping tablet replica
I20260927 13:59:54.040269 6729 raft_consensus.cc:2229] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:59:54.040892 6729 raft_consensus.cc:2258] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:59:54.043560 6729 ts_tablet_manager.cc:1905] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:59:54.051724 6728 tablet_replica.cc:330] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b: stopping tablet replica
I20260927 13:59:54.052204 6728 raft_consensus.cc:2229] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:59:54.053018 6728 raft_consensus.cc:2258] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:59:54.054179 6729 ts_tablet_manager.cc:1918] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:59:54.054456 6729 log.cc:1196] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/90b7294381da4370a2786b5af0206bb9
I20260927 13:59:54.055106 6729 ts_tablet_manager.cc:1939] T 90b7294381da4370a2786b5af0206bb9 P 3b3d36d518f14027ba4f7c458fe7b8fb: Deleting consensus metadata
I20260927 13:59:54.055261 6728 ts_tablet_manager.cc:1905] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:59:54.057163 6228 catalog_manager.cc:4841] TS 3b3d36d518f14027ba4f7c458fe7b8fb (127.22.26.193:33815): tablet 90b7294381da4370a2786b5af0206bb9 (table test-table [id=b350706e990240c7a48fc4acc44920df]) successfully deleted
I20260927 13:59:54.058068 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:54.031647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.22.26.193:33815)
0927 13:59:54.031883 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:54.031892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:54.031913 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:59:54.031990 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:54.031997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:54.032106 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:54.032223 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:54.034908 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:54.036541 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:54.040102 (+ 3561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:54.040853 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:54.041013 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:54.048001 (+ 6988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:54.048015 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:54.048037 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:54.048079 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:54.052286 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:54.053411 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:54.053420 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:54.053424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:54.053581 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:54.055017 (+ 1436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:54.055023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:54.055029 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:54.055589 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:54.055886 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:54.057647 (+ 1761us) server_negotiation.cc:299] Negotiation successful
0927 13:59:54.057816 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 13:59:54.058981 6421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:54.062726 6728 ts_tablet_manager.cc:1918] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:59:54.062963 6728 log.cc:1196] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/90b7294381da4370a2786b5af0206bb9
I20260927 13:59:54.063601 6728 ts_tablet_manager.cc:1939] T 90b7294381da4370a2786b5af0206bb9 P ab7c5f711c9b4dd4b1c4b8b376116d5b: Deleting consensus metadata
I20260927 13:59:54.065644 6226 catalog_manager.cc:4841] TS ab7c5f711c9b4dd4b1c4b8b376116d5b (127.22.26.195:39767): tablet 90b7294381da4370a2786b5af0206bb9 (table test-table [id=b350706e990240c7a48fc4acc44920df]) successfully deleted
I20260927 13:59:54.067620 6730 tablet_replica.cc:330] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58: stopping tablet replica
I20260927 13:59:54.068199 6730 raft_consensus.cc:2229] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:59:54.068747 6730 raft_consensus.cc:2258] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:59:54.071213 6730 ts_tablet_manager.cc:1905] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:59:54.078609 6730 ts_tablet_manager.cc:1918] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:59:54.078850 6730 log.cc:1196] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/90b7294381da4370a2786b5af0206bb9
I20260927 13:59:54.079484 6730 ts_tablet_manager.cc:1939] T 90b7294381da4370a2786b5af0206bb9 P a158484037994d5eb0ef02d9bbd87a58: Deleting consensus metadata
I20260927 13:59:54.081405 6228 catalog_manager.cc:4841] TS a158484037994d5eb0ef02d9bbd87a58 (127.22.26.194:42591): tablet 90b7294381da4370a2786b5af0206bb9 (table test-table [id=b350706e990240c7a48fc4acc44920df]) successfully deleted
I20260927 13:59:54.159137 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 6278
I20260927 13:59:54.178136 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 6430
I20260927 13:59:54.195959 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:54.178979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.22.26.194:42591)
0927 13:59:54.179211 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:54.179216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:54.179234 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:59:54.179456 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:54.179460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:54.179533 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:54.179671 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:54.181392 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:54.183511 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:54.185403 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:54.185780 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:54.185877 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:54.189466 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:54.189481 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:54.189497 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:54.189554 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:54.193136 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:54.193989 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:54.193996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:54.193999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:54.194145 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:54.194570 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:54.194573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:54.194575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:54.194929 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:54.195250 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:54.195520 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 13:59:54.195684 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:59:54.196610 6553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:54.286502 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 6430
I20260927 13:59:54.303644 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 6561
I20260927 13:59:54.320233 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:59:54.304477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.22.26.195:39767)
0927 13:59:54.304684 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 13:59:54.304690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:59:54.304712 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:59:54.304839 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:59:54.304844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:59:54.304928 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:59:54.305077 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:59:54.306526 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:54.307829 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:54.309265 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:59:54.309989 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:59:54.310132 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:59:54.312788 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:59:54.312807 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:59:54.312826 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:59:54.312878 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:59:54.317037 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:54.317740 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:54.317750 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:54.317754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:54.317955 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:59:54.318336 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:59:54.318341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:59:54.318344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:59:54.318804 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:59:54.319188 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0927 13:59:54.319478 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 13:59:54.319738 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 13:59:54.321025 6684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:54.443778 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 6561
I20260927 13:59:54.463732 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 6208
I20260927 13:59:54.464866 6271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:59:54.581012 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 6208
I20260927 14:00:14.603335 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42401
--webserver_interface=127.22.26.254
--webserver_port=46199
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:42401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:14.940809 6735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:14.941108 6735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:14.941258 6735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:14.948464 6735 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 14:00:14.948553 6735 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 14:00:14.948597 6735 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 14:00:14.948701 6735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:00:14.948793 6735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:14.948909 6735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:14.948971 6735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:00:14.949039 6735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:00:14.958648 6735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:42401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=46199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:14.959920 6735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:14.961238 6735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:14.973882 6741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:14.973897 6743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:14.974040 6740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:14.974608 6735 server_base.cc:1034] running on GCE node
I20260927 14:00:14.975648 6735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:14.977164 6735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:14.978395 6735 hybrid_clock.cc:648] HybridClock initialized: now 1790517614978323 us; error 38 us; skew 500 ppm
I20260927 14:00:14.978788 6735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:14 dist-test-slave-gkks krb5kdc[5657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517614, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:14.982262 6735 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 14:00:14.983906 6735 webserver.cc:460] Webserver started at http://127.22.26.254:46199/ using document root <none> and password file <none>
I20260927 14:00:14.984635 6735 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:14.984730 6735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:14.990597 6735 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 14:00:14.994784 6750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:14.996400 6735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 14:00:14.996649 6735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "d803bc2147c048cf8be3d45fd18fb911"
format_stamp: "Formatted at 2026-09-27 13:59:29 on dist-test-slave-gkks"
server_key: "4efdc8b9c3f39a9fdde08ab6e7ed1217"
server_key_iv: "3579a200d8c4b6115c3af4b46a0026fe"
server_key_version: "encryptionkey@0"
I20260927 14:00:14.997718 6735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:15.049093 6735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:15.053692 6735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:15.054006 6735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:15.073119 6735 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:42401
I20260927 14:00:15.073223 6801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:42401 every 8 connection(s)
I20260927 14:00:15.074683 6735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:00:15.080035 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 6735
I20260927 14:00:15.081039 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:33815
--local_ip_for_outbound_sockets=127.22.26.193
--tserver_master_addrs=127.22.26.254:42401
--webserver_port=38635
--webserver_interface=127.22.26.193
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:00:15.082198 6802 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:00:15.087509 6802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Bootstrap starting.
I20260927 14:00:15.109534 6802 log.cc:824] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:15.125797 6802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:00:15.126346 6802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Bootstrap complete.
I20260927 14:00:15.135649 6802 raft_consensus.cc:348] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } }
I20260927 14:00:15.136549 6802 raft_consensus.cc:729] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d803bc2147c048cf8be3d45fd18fb911, State: Initialized, Role: FOLLOWER
I20260927 14:00:15.137264 6802 consensus_queue.cc:260] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } }
I20260927 14:00:15.137494 6802 raft_consensus.cc:388] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:00:15.137605 6802 raft_consensus.cc:482] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:00:15.137797 6802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 2 FOLLOWER]: Advancing to term 3
I20260927 14:00:15.141621 6802 raft_consensus.cc:504] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } }
I20260927 14:00:15.142127 6802 leader_election.cc:302] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d803bc2147c048cf8be3d45fd18fb911; no voters:
I20260927 14:00:15.142854 6802 leader_election.cc:288] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 14:00:15.142989 6806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 14:00:15.144248 6806 raft_consensus.cc:686] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [term 3 LEADER]: Becoming Leader. State: Replica: d803bc2147c048cf8be3d45fd18fb911, State: Running, Role: LEADER
I20260927 14:00:15.144877 6806 consensus_queue.cc:237] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } }
I20260927 14:00:15.145177 6802 sys_catalog.cc:564] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:00:15.147284 6808 sys_catalog.cc:455] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d803bc2147c048cf8be3d45fd18fb911. Latest consensus state: current_term: 3 leader_uuid: "d803bc2147c048cf8be3d45fd18fb911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } } }
I20260927 14:00:15.147686 6808 sys_catalog.cc:458] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:15.148242 6807 sys_catalog.cc:455] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d803bc2147c048cf8be3d45fd18fb911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d803bc2147c048cf8be3d45fd18fb911" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42401 } } }
I20260927 14:00:15.148532 6807 sys_catalog.cc:458] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:15.150909 6814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:00:15.159825 6814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:00:15.161273 6814 catalog_manager.cc:1259] Loaded cluster ID: 4cab217f3b4e43bcb9b65e13c688fd44
I20260927 14:00:15.161417 6814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:00:15.165596 6814 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:00:15.167361 6814 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Loaded TSK: 0
I20260927 14:00:15.167579 6814 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d803bc2147c048cf8be3d45fd18fb911: Loaded TSK: 1
F20260927 14:00:15.168776 6814 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f43ba4589ad google::LogMessage::Fail() at ??:0
@ 0x7f43ba45b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f43ba4584de google::LogMessage::Flush() at ??:0
@ 0x7f43ba459a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f43be1bc7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f43be1ab73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f43cfe8b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f43cfe8a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f43cff162a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f43cff160c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f43cff605e9 std::function<>::operator()() at ??:0
@ 0x7f43cfe8ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f43cfe8cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f43cff14610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f43cff14439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f43bd0a4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f43bd0a791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f43bd0a7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f43bd082eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f43c035c6db start_thread at ??:0
@ 0x7f43b92f261f clone at ??:0
W20260927 14:00:15.465126 6804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:15.465437 6804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:15.465602 6804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:15.472863 6804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:15.473007 6804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:15.473174 6804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 14:00:15.482759 6804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:33815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=38635
--tserver_master_addrs=127.22.26.254:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:15.484002 6804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:15.485316 6804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:15.498255 6850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:15.498346 6849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:15.498319 6852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:15.499190 6804 server_base.cc:1034] running on GCE node
I20260927 14:00:15.499804 6804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:15.500670 6804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:15.501904 6804 hybrid_clock.cc:648] HybridClock initialized: now 1790517615501856 us; error 43 us; skew 500 ppm
I20260927 14:00:15.502205 6804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:15 dist-test-slave-gkks krb5kdc[5657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517615, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:15.505393 6804 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 14:00:15.506731 6804 webserver.cc:460] Webserver started at http://127.22.26.193:38635/ using document root <none> and password file <none>
I20260927 14:00:15.507453 6804 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:15.507596 6804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:15.512518 6804 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 14:00:15.515196 6859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:15.516444 6804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 14:00:15.516614 6804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "3b3d36d518f14027ba4f7c458fe7b8fb"
format_stamp: "Formatted at 2026-09-27 13:59:29 on dist-test-slave-gkks"
server_key: "93efd19c5d88f27f12f4116f9c11f58c"
server_key_iv: "35ab40ae05d64831c3d14d7c53f45cd5"
server_key_version: "encryptionkey@0"
I20260927 14:00:15.517562 6804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:15.555563 6804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:15.560037 6804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:15.560328 6804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:15.561401 6804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:15.562779 6804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:15.562903 6804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:15.563036 6804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:15.563139 6804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:15.596728 6804 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:33815
I20260927 14:00:15.596814 6971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:33815 every 8 connection(s)
I20260927 14:00:15.598270 6804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 14:00:15.600126 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 6804
I20260927 14:00:15.600809 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:42591
--local_ip_for_outbound_sockets=127.22.26.194
--tserver_master_addrs=127.22.26.254:42401
--webserver_port=41081
--webserver_interface=127.22.26.194
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:00:15.604117 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:15.601790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:46997)
0927 14:00:15.602651 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:15.602961 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":318,"threads_started":1}
W20260927 14:00:15.604907 6972 heartbeater.cc:644] Failed to heartbeat to 127.22.26.254:42401 (0 consecutive failures): Network error: Failed to ping master at 127.22.26.254:42401: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
I20260927 14:00:15.605821 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:15.605447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:59555)
0927 14:00:15.605630 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:15.605710 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 14:00:15.606769 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:15.606429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:47731)
0927 14:00:15.606581 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:15.606665 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 14:00:15.607069 6972 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:00:15.926193 6975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:15.926456 6975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:15.926558 6975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:15.933606 6975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:15.933709 6975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:15.933820 6975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 14:00:15.943439 6975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:42591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=41081
--tserver_master_addrs=127.22.26.254:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:15.944641 6975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:15.945854 6975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:15.958592 6980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:15.958544 6983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:15.958501 6981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:15.959749 6975 server_base.cc:1034] running on GCE node
I20260927 14:00:15.960321 6975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:15.961272 6975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:15.962507 6975 hybrid_clock.cc:648] HybridClock initialized: now 1790517615962457 us; error 45 us; skew 500 ppm
I20260927 14:00:15.962824 6975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:15 dist-test-slave-gkks krb5kdc[5657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517615, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:15.965880 6975 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 14:00:15.967315 6975 webserver.cc:460] Webserver started at http://127.22.26.194:41081/ using document root <none> and password file <none>
I20260927 14:00:15.967943 6975 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:15.968041 6975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:15.973217 6975 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 14:00:15.975911 6990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:15.977180 6975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:00:15.977356 6975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "a158484037994d5eb0ef02d9bbd87a58"
format_stamp: "Formatted at 2026-09-27 13:59:30 on dist-test-slave-gkks"
server_key: "8bb3008b081bbce8a1d3a997b2b2c89d"
server_key_iv: "17d254fbf9863145a3f32d0406ce9892"
server_key_version: "encryptionkey@0"
I20260927 14:00:15.978312 6975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:16.008008 6975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:16.012303 6975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:16.012607 6975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:16.013697 6975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:16.015261 6975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:16.015393 6975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:16.015528 6975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:16.015604 6975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:16.050889 6975 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:42591
I20260927 14:00:16.051004 7102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:42591 every 8 connection(s)
I20260927 14:00:16.052412 6975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 14:00:16.056619 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 6975
I20260927 14:00:16.057426 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:39767
--local_ip_for_outbound_sockets=127.22.26.195
--tserver_master_addrs=127.22.26.254:42401
--webserver_port=42005
--webserver_interface=127.22.26.195
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:00:16.057817 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:16.055480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:36773)
0927 14:00:16.056308 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:16.056616 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":268,"threads_started":1}
W20260927 14:00:16.058532 7103 heartbeater.cc:644] Failed to heartbeat to 127.22.26.254:42401 (0 consecutive failures): Network error: Failed to ping master at 127.22.26.254:42401: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
I20260927 14:00:16.059506 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:16.059051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:33113)
0927 14:00:16.059260 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:16.059354 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 14:00:16.060473 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:16.060044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:46869)
0927 14:00:16.060218 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:16.060316 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 14:00:16.060719 7103 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:00:16.387455 7106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:16.387749 7106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:16.387912 7106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:16.395341 7106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:16.395467 7106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:16.395663 7106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 14:00:16.405223 7106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:39767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=42005
--tserver_master_addrs=127.22.26.254:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:16.406482 7106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:16.407855 7106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:16.420650 7111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:16.420676 7112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:16.420732 7114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:16.421486 7106 server_base.cc:1034] running on GCE node
I20260927 14:00:16.422184 7106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:16.423131 7106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:16.424360 7106 hybrid_clock.cc:648] HybridClock initialized: now 1790517616424305 us; error 51 us; skew 500 ppm
I20260927 14:00:16.424664 7106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:16 dist-test-slave-gkks krb5kdc[5657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517616, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:16.427769 7106 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 14:00:16.429193 7106 webserver.cc:460] Webserver started at http://127.22.26.195:42005/ using document root <none> and password file <none>
I20260927 14:00:16.429840 7106 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:16.429980 7106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:16.435040 7106 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 14:00:16.437773 7121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:16.439029 7106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:00:16.439230 7106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "ab7c5f711c9b4dd4b1c4b8b376116d5b"
format_stamp: "Formatted at 2026-09-27 13:59:30 on dist-test-slave-gkks"
server_key: "a0c0db4fb5342bec71fef268caea9f15"
server_key_iv: "01202bb36e0d0f92413262245a0889f9"
server_key_version: "encryptionkey@0"
I20260927 14:00:16.440163 7106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:16.453454 7106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:16.458092 7106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:16.458400 7106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:16.459501 7106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:16.460953 7106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:16.461091 7106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:16.461239 7106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:16.461337 7106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:16.497527 7106 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:39767
I20260927 14:00:16.497588 7233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:39767 every 8 connection(s)
I20260927 14:00:16.499214 7106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 14:00:16.501490 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 7106
I20260927 14:00:16.504485 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:16.502323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:34749)
0927 14:00:16.503272 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:16.503579 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":380,"threads_started":1}
W20260927 14:00:16.505158 7234 heartbeater.cc:644] Failed to heartbeat to 127.22.26.254:42401 (0 consecutive failures): Network error: Failed to ping master at 127.22.26.254:42401: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
I20260927 14:00:16.506023 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:16.505683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:36535)
0927 14:00:16.505843 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:16.505919 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 14:00:16.506923 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:16.506577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:45885)
0927 14:00:16.506726 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:16.506811 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 14:00:16.507196 7234 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 14:00:16.608556 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:16.607632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:56465)
0927 14:00:16.608144 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:16.608289 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":246,"threads_started":1}
I20260927 14:00:17.062052 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:17.061203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:57703)
0927 14:00:17.061717 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:17.061810 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":246,"threads_started":1}
I20260927 14:00:17.508474 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:17.507693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:33221)
0927 14:00:17.508124 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:17.508242 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":207,"threads_started":1}
I20260927 14:00:17.610261 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:17.609538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:41039)
0927 14:00:17.610013 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:17.610099 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":238,"threads_started":1}
I20260927 14:00:18.064189 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:18.063205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:48919)
0927 14:00:18.063913 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:18.064014 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":460,"threads_started":1}
I20260927 14:00:18.510102 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:18.509409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:52729)
0927 14:00:18.509866 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:18.509943 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":225,"threads_started":1}
I20260927 14:00:18.611994 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:18.611303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:56707)
0927 14:00:18.611756 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:18.611832 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":248,"threads_started":1}
I20260927 14:00:19.065734 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:19.065047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:44665)
0927 14:00:19.065483 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:19.065564 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":237,"threads_started":1}
I20260927 14:00:19.511626 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:19.510928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:60635)
0927 14:00:19.511385 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:19.511462 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":266,"threads_started":1}
I20260927 14:00:19.613598 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:19.612844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:45991)
0927 14:00:19.613347 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:19.613428 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":291,"threads_started":1}
I20260927 14:00:20.067625 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:20.066837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:56291)
0927 14:00:20.067385 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:20.067461 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":337,"threads_started":1}
I20260927 14:00:20.513255 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:20.512426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:34405)
0927 14:00:20.512967 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:20.513060 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":201,"threads_started":1}
I20260927 14:00:20.615221 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:20.614478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:55951)
0927 14:00:20.614900 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:20.614985 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":223,"threads_started":1}
I20260927 14:00:21.069371 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:21.068601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:51967)
0927 14:00:21.069086 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:21.069185 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":254,"threads_started":1}
I20260927 14:00:21.515038 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:21.514313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:41447)
0927 14:00:21.514787 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:21.514869 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":234,"threads_started":1}
I20260927 14:00:21.617172 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:21.616143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:57899)
0927 14:00:21.616757 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:21.616923 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":336,"threads_started":1}
I20260927 14:00:22.071180 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:22.070418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:36185)
0927 14:00:22.070864 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:22.070955 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260927 14:00:22.516666 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:22.515972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:46581)
0927 14:00:22.516407 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:22.516501 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":211,"threads_started":1}
I20260927 14:00:22.618940 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:22.618132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:39297)
0927 14:00:22.618626 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:22.618719 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":214,"threads_started":1}
I20260927 14:00:23.072956 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:23.072097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:55603)
0927 14:00:23.072590 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:23.072718 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":215,"threads_started":1}
I20260927 14:00:23.518540 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:23.517720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:49789)
0927 14:00:23.518233 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:23.518323 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":224,"threads_started":1}
I20260927 14:00:23.620738 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:23.620017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:48303)
0927 14:00:23.620439 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:23.620539 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":214,"threads_started":1}
I20260927 14:00:24.074735 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:24.074023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:60871)
0927 14:00:24.074495 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:24.074577 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":225,"threads_started":1}
I20260927 14:00:24.520431 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:24.519578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:60543)
0927 14:00:24.520123 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:24.520225 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":253,"threads_started":1}
I20260927 14:00:24.622347 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:24.621657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:60241)
0927 14:00:24.622091 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:24.622182 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":209,"threads_started":1}
I20260927 14:00:25.076432 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:25.075632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:58797)
0927 14:00:25.076144 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:25.076251 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":266,"threads_started":1}
I20260927 14:00:25.522307 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:25.521480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:59239)
0927 14:00:25.522049 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:25.522133 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":252,"threads_started":1}
I20260927 14:00:25.624050 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:25.623334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:45717)
0927 14:00:25.623815 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:25.623895 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":270,"threads_started":1}
I20260927 14:00:26.078159 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:26.077379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:44477)
0927 14:00:26.077930 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:26.078004 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":345,"threads_started":1}
I20260927 14:00:26.523947 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:26.523278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:40389)
0927 14:00:26.523704 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:26.523783 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":224,"threads_started":1}
I20260927 14:00:26.625705 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:26.624978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:44553)
0927 14:00:26.625406 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:26.625496 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":209,"threads_started":1}
I20260927 14:00:27.079799 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:27.079051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:39619)
0927 14:00:27.079508 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:27.079600 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":202,"threads_started":1}
I20260927 14:00:27.525599 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:27.524898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:55545)
0927 14:00:27.525333 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:27.525418 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":215,"threads_started":1}
I20260927 14:00:27.627364 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:27.626661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:51273)
0927 14:00:27.627126 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:27.627208 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":221,"threads_started":1}
I20260927 14:00:28.081581 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:28.080786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:56301)
0927 14:00:28.081207 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:28.081300 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":218,"threads_started":1}
I20260927 14:00:28.527240 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:28.526509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:44483)
0927 14:00:28.526937 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:28.527012 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":230,"threads_started":1}
I20260927 14:00:28.629215 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:28.628303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:37137)
0927 14:00:28.628862 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:28.628981 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":291,"threads_started":1}
I20260927 14:00:29.083406 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:29.082571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:38201)
0927 14:00:29.083034 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:29.083216 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":217,"threads_started":1}
I20260927 14:00:29.528893 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:29.528061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:47753)
0927 14:00:29.528543 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:29.528634 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":234,"threads_started":1}
I20260927 14:00:29.630720 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:29.629996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:39355)
0927 14:00:29.630440 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:29.630521 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":230,"threads_started":1}
I20260927 14:00:30.085027 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:30.084305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:41521)
0927 14:00:30.084763 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:30.084845 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260927 14:00:30.530510 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:30.529730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:37605)
0927 14:00:30.530209 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:30.530309 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":238,"threads_started":1}
I20260927 14:00:30.632539 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:30.631826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.193:48809)
0927 14:00:30.632268 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:30.632356 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":220,"threads_started":1}
I20260927 14:00:31.086884 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:31.086123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.194:56743)
0927 14:00:31.086593 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:31.086694 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260927 14:00:31.504208 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 6804
I20260927 14:00:31.531139 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:31.505390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.22.26.193:33815)
0927 14:00:31.506063 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:31.506087 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:31.506145 (+ 58us) server_negotiation.cc:407] Connection header received
0927 14:00:31.508130 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:31.508140 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:31.508609 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:31.508910 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:31.510333 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:31.511826 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:31.513069 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:31.513773 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:31.513911 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:31.525080 (+ 11169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:31.525096 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:31.525113 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:31.525167 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:31.528060 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:31.528760 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:31.528767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:31.528774 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:31.528931 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:31.529255 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:31.529258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:31.529260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:31.529782 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:31.530056 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:31.530386 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 14:00:31.530794 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":294,"threads_started":1}
I20260927 14:00:31.532413 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:31.531736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:42401 (local address 127.22.26.195:39865)
0927 14:00:31.532163 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:31.532250 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.254:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":219,"threads_started":1}
I20260927 14:00:31.532768 6967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:31.618217 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 6804
I20260927 14:00:31.634900 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 6975
I20260927 14:00:31.653188 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:31.635922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.22.26.194:42591)
0927 14:00:31.636377 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:31.636392 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:31.636428 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:00:31.637949 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:31.637958 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:31.638372 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:31.638668 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:31.640018 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:31.641268 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:31.642718 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:31.643625 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:31.643755 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:31.646515 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:31.646523 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:31.646547 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:31.646589 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:31.649786 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:31.650766 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:31.650775 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:31.650797 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:31.651040 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:31.651432 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:31.651435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:31.651437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:31.652022 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:31.652250 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:31.652594 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 14:00:31.652902 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":197,"threads_started":1}
I20260927 14:00:31.654579 7098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:31.739368 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 6975
I20260927 14:00:31.756814 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 7106
I20260927 14:00:31.774345 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:31.757854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46398 (local address 127.22.26.195:39767)
0927 14:00:31.758407 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:31.758422 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:31.758463 (+ 41us) server_negotiation.cc:407] Connection header received
0927 14:00:31.760154 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:31.760164 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:31.760600 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:31.760939 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:31.762365 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:31.763714 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:31.764930 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:31.765589 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:31.765721 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:31.768355 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:31.768363 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:31.768377 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:31.768416 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:31.771599 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:31.772202 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:31.772208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:31.772214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:31.772371 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:31.772694 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:31.772697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:31.772699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:31.773192 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:31.773406 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:31.773731 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 14:00:31.774053 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":246,"threads_started":1}
I20260927 14:00:31.775751 7229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:31.858743 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 7106
2026-09-27T14:00:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64293 ms)
[----------] 30 tests from SecurityITest (301010 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:00:31 dist-test-slave-gkks krb5kdc[7291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:00:31 dist-test-slave-gkks krb5kdc[7291](info): set up 2 sockets
Sep 27 14:00:31 dist-test-slave-gkks krb5kdc[7291](info): commencing operation
krb5kdc: starting...
W20260927 14:00:32.995642 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:00:33 dist-test-slave-gkks krb5kdc[7291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:00:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:00:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 14:00:33.251008 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39411
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:39411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:33.604058 7304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:33.604346 7304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:33.604539 7304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:33.612317 7304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:00:33.612425 7304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:33.612545 7304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:33.612641 7304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:00:33.612710 7304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:00:33.622614 7304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:39411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:39411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:33.623992 7304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:33.625344 7304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:33.637724 7309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:33.637665 7312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:33.637710 7310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:33.638065 7304 server_base.cc:1034] running on GCE node
I20260927 14:00:33.639221 7304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:33.640630 7304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:33.641836 7304 hybrid_clock.cc:648] HybridClock initialized: now 1790517633641793 us; error 32 us; skew 500 ppm
I20260927 14:00:33.642196 7304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:33 dist-test-slave-gkks krb5kdc[7291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517633, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:33.645371 7304 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 14:00:33.646822 7304 webserver.cc:460] Webserver started at http://127.22.26.254:39629/ using document root <none> and password file <none>
I20260927 14:00:33.647553 7304 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:33.647702 7304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:33.648068 7304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:33.651017 7304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "555c0f7eea434dfc9289c2b978b95ce3"
format_stamp: "Formatted at 2026-09-27 14:00:33 on dist-test-slave-gkks"
server_key: "ca5bb306fca92968081e5ab09c958b67"
server_key_iv: "1d752390162522d15bcdf6a7746384a8"
server_key_version: "encryptionkey@0"
I20260927 14:00:33.651839 7304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "555c0f7eea434dfc9289c2b978b95ce3"
format_stamp: "Formatted at 2026-09-27 14:00:33 on dist-test-slave-gkks"
server_key: "ca5bb306fca92968081e5ab09c958b67"
server_key_iv: "1d752390162522d15bcdf6a7746384a8"
server_key_version: "encryptionkey@0"
I20260927 14:00:33.656880 7304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 14:00:33.660300 7319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:33.661773 7304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:00:33.662060 7304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "555c0f7eea434dfc9289c2b978b95ce3"
format_stamp: "Formatted at 2026-09-27 14:00:33 on dist-test-slave-gkks"
server_key: "ca5bb306fca92968081e5ab09c958b67"
server_key_iv: "1d752390162522d15bcdf6a7746384a8"
server_key_version: "encryptionkey@0"
I20260927 14:00:33.662238 7304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:33.686398 7304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:33.691210 7304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:33.691525 7304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:33.710886 7304 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:39411
I20260927 14:00:33.710904 7370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:39411 every 8 connection(s)
I20260927 14:00:33.712353 7304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:00:33.715140 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 7304
I20260927 14:00:33.715360 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:00:33.716014 22635 external_mini_cluster.cc:1428] Setting key e071992cd68303422234709ab6bfa14d
I20260927 14:00:33.716562 7371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:33.731771 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3: Bootstrap starting.
Sep 27 14:00:33 dist-test-slave-gkks krb5kdc[7291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:00:33.735152 7371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:33.736238 7371 log.cc:824] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:33.739869 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3: No bootstrap required, opened a new log
I20260927 14:00:33.741050 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:33.718108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.22.26.254:39411)
0927 14:00:33.718961 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:33.718973 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:33.719014 (+ 41us) server_negotiation.cc:407] Connection header received
0927 14:00:33.721342 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:33.721367 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:33.721797 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:33.722149 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:33.724113 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:33.725448 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:33.727377 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:33.728080 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:33.728247 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:33.733197 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:33.733205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:33.733222 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:33.733268 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:33.736473 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:33.737173 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:33.737182 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:33.737188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:33.737420 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:33.738001 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:33.738010 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:33.738015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:33.738674 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:33.738907 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:33.739452 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 14:00:33.739918 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":385,"threads_started":1}
I20260927 14:00:33.748770 7371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555c0f7eea434dfc9289c2b978b95ce3" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39411 } }
I20260927 14:00:33.749161 7371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:33.749269 7371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555c0f7eea434dfc9289c2b978b95ce3, State: Initialized, Role: FOLLOWER
I20260927 14:00:33.749799 7371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555c0f7eea434dfc9289c2b978b95ce3" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39411 } }
I20260927 14:00:33.750036 7371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:00:33.750190 7371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:00:33.750361 7371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:33.752806 7371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555c0f7eea434dfc9289c2b978b95ce3" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39411 } }
I20260927 14:00:33.753314 7371 leader_election.cc:302] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 555c0f7eea434dfc9289c2b978b95ce3; no voters:
I20260927 14:00:33.753934 7371 leader_election.cc:288] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:00:33.754117 7376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:33.755216 7376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [term 1 LEADER]: Becoming Leader. State: Replica: 555c0f7eea434dfc9289c2b978b95ce3, State: Running, Role: LEADER
I20260927 14:00:33.755738 7376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555c0f7eea434dfc9289c2b978b95ce3" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39411 } }
I20260927 14:00:33.755944 7371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:00:33.759250 7378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 555c0f7eea434dfc9289c2b978b95ce3. Latest consensus state: current_term: 1 leader_uuid: "555c0f7eea434dfc9289c2b978b95ce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555c0f7eea434dfc9289c2b978b95ce3" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39411 } } }
I20260927 14:00:33.759932 7378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:33.760560 7377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "555c0f7eea434dfc9289c2b978b95ce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555c0f7eea434dfc9289c2b978b95ce3" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 39411 } } }
I20260927 14:00:33.760926 7377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:33.764637 7385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:00:33.769337 7385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:00:33.778083 7385 catalog_manager.cc:1347] Generated new cluster ID: 2f5aa1aa77de40bca0222cfba3c02a86
I20260927 14:00:33.778206 7385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:00:33.798796 7385 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:00:33.800371 7385 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:00:33.817211 7385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3: Generated new TSK 0
I20260927 14:00:33.818195 7385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 14:00:33.877033 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39411
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:34.226898 7397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:34.227276 7397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:34.227388 7397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:34.235549 7397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:34.235661 7397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:34.235791 7397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 14:00:34.246814 7397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:34.248270 7397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:34.249620 7397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:34.263752 7402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:34.263829 7405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:34.263794 7403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:34.265143 7397 server_base.cc:1034] running on GCE node
I20260927 14:00:34.265802 7397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:34.266881 7397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:34.268119 7397 hybrid_clock.cc:648] HybridClock initialized: now 1790517634268060 us; error 51 us; skew 500 ppm
I20260927 14:00:34.268497 7397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:34 dist-test-slave-gkks krb5kdc[7291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517634, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:34.271756 7397 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 14:00:34.273388 7397 webserver.cc:460] Webserver started at http://127.22.26.193:42743/ using document root <none> and password file <none>
I20260927 14:00:34.274098 7397 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:34.274199 7397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:34.274513 7397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:34.277506 7397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86"
format_stamp: "Formatted at 2026-09-27 14:00:34 on dist-test-slave-gkks"
server_key: "e987129fc5e619d8be5e7cd44ebb9c3d"
server_key_iv: "7a097e1d32fad7c1e94ad7700f413768"
server_key_version: "encryptionkey@0"
I20260927 14:00:34.278272 7397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86"
format_stamp: "Formatted at 2026-09-27 14:00:34 on dist-test-slave-gkks"
server_key: "e987129fc5e619d8be5e7cd44ebb9c3d"
server_key_iv: "7a097e1d32fad7c1e94ad7700f413768"
server_key_version: "encryptionkey@0"
I20260927 14:00:34.283675 7397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 14:00:34.287258 7412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:34.288853 7397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:00:34.289112 7397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86"
format_stamp: "Formatted at 2026-09-27 14:00:34 on dist-test-slave-gkks"
server_key: "e987129fc5e619d8be5e7cd44ebb9c3d"
server_key_iv: "7a097e1d32fad7c1e94ad7700f413768"
server_key_version: "encryptionkey@0"
I20260927 14:00:34.289415 7397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:34.322351 7397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:34.327373 7397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:34.327736 7397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:34.328964 7397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:34.330402 7397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:34.330472 7397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:34.330621 7397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:34.330704 7397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:34.364372 7397 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:33073
I20260927 14:00:34.364413 7524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:33073 every 8 connection(s)
I20260927 14:00:34.365800 7397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 14:00:34.372594 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 7397
I20260927 14:00:34.372826 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 14:00:34.373524 22635 external_mini_cluster.cc:1428] Setting key c3ad38b5efcc33f2947456fe6491b617
Sep 27 14:00:34 dist-test-slave-gkks krb5kdc[7291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517634, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:00:34.389137 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:34.368656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:51305 (local address 127.22.26.254:39411)
0927 14:00:34.369108 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:34.369113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:34.370048 (+ 935us) server_negotiation.cc:407] Connection header received
0927 14:00:34.372159 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:34.372165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:34.372267 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:34.372399 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:34.374287 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:34.375199 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:34.377074 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:34.377821 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:34.377965 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:34.382114 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:34.382124 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:34.382132 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:34.382200 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:34.385476 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:34.386548 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:34.386552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:34.386554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:34.386677 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:34.387231 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:34.387234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:34.387236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:34.387517 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:34.387639 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:34.388690 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0927 14:00:34.388872 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":241,"threads_started":1}
I20260927 14:00:34.390750 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:34.369099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39411 (local address 127.22.26.193:51305)
0927 14:00:34.369874 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:34.369919 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:34.371863 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:34.372657 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:34.372667 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:34.373299 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:34.374050 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:34.374109 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:34.375421 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:34.375427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:34.376916 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:34.376929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:34.378046 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:34.378052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:34.378279 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:34.379025 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:00:34.379049 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:34.381844 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:34.385638 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:34.385652 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:34.385673 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:34.386408 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:34.386796 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:34.386803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:34.386810 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:34.387112 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:34.387667 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:34.387679 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:34.388198 (+ 519us) client_negotiation.cc:769] Sending connection context
0927 14:00:34.388546 (+ 348us) client_negotiation.cc:240] Negotiation successful
0927 14:00:34.389175 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":302,"threads_started":1}
I20260927 14:00:34.392534 7525 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39411
I20260927 14:00:34.392856 7525 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:34.393714 7525 heartbeater.cc:511] Master 127.22.26.254:39411 requested a full tablet report, sending...
I20260927 14:00:34.396209 7336 ts_manager.cc:194] Registered new tserver with Master: 55e6a2ac4aea4f53a70d43bdad1b1c86 (127.22.26.193:33073)
I20260927 14:00:34.398589 7336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:51305
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 14:00:34.434340 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39411
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:34.760797 7531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:34.761128 7531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:34.761312 7531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:34.769068 7531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:34.769181 7531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:34.769375 7531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 14:00:34.779302 7531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:34.780627 7531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:34.781962 7531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:34.795325 7536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:34.795243 7537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:34.795859 7539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:34.796329 7531 server_base.cc:1034] running on GCE node
I20260927 14:00:34.796897 7531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:34.797863 7531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:34.799120 7531 hybrid_clock.cc:648] HybridClock initialized: now 1790517634799044 us; error 37 us; skew 500 ppm
I20260927 14:00:34.799489 7531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:34 dist-test-slave-gkks krb5kdc[7291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517634, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:34.802628 7531 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 14:00:34.804129 7531 webserver.cc:460] Webserver started at http://127.22.26.194:44563/ using document root <none> and password file <none>
I20260927 14:00:34.804862 7531 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:34.805018 7531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:34.805362 7531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:34.808840 7531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "e775f523ab424156b457c3866f485d6c"
format_stamp: "Formatted at 2026-09-27 14:00:34 on dist-test-slave-gkks"
server_key: "5196135467c84f173df254094e73b408"
server_key_iv: "46b9bf965ebf97039f6530f5a07351ff"
server_key_version: "encryptionkey@0"
I20260927 14:00:34.809753 7531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "e775f523ab424156b457c3866f485d6c"
format_stamp: "Formatted at 2026-09-27 14:00:34 on dist-test-slave-gkks"
server_key: "5196135467c84f173df254094e73b408"
server_key_iv: "46b9bf965ebf97039f6530f5a07351ff"
server_key_version: "encryptionkey@0"
I20260927 14:00:34.815372 7531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 14:00:34.819164 7546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:34.820578 7531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:00:34.820785 7531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "e775f523ab424156b457c3866f485d6c"
format_stamp: "Formatted at 2026-09-27 14:00:34 on dist-test-slave-gkks"
server_key: "5196135467c84f173df254094e73b408"
server_key_iv: "46b9bf965ebf97039f6530f5a07351ff"
server_key_version: "encryptionkey@0"
I20260927 14:00:34.821002 7531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:34.832886 7531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:34.837365 7531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:34.837664 7531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:34.838716 7531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:34.840104 7531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:34.840206 7531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:34.840348 7531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:34.840420 7531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:34.874222 7531 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:40737
I20260927 14:00:34.874303 7658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:40737 every 8 connection(s)
I20260927 14:00:34.875608 7531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 14:00:34.879241 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 7531
I20260927 14:00:34.879463 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 14:00:34.880105 22635 external_mini_cluster.cc:1428] Setting key 7bbc397e4de2653d17d87e2364599e22
Sep 27 14:00:34 dist-test-slave-gkks krb5kdc[7291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517634, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:00:34.898907 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:34.878244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:35463 (local address 127.22.26.254:39411)
0927 14:00:34.878428 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:34.878432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:34.879866 (+ 1434us) server_negotiation.cc:407] Connection header received
0927 14:00:34.882515 (+ 2649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:34.882519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:34.882590 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:34.882675 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:34.884958 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:34.885690 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:34.888198 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:34.888658 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:34.888768 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:34.893019 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:34.893024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:34.893027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:34.893082 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:34.895817 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:34.896568 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:34.896572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:34.896575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:34.896701 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:34.897134 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:34.897144 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:34.897146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:34.897422 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:34.897593 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:34.898306 (+ 713us) server_negotiation.cc:299] Negotiation successful
0927 14:00:34.898542 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 14:00:34.900094 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:34.878713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39411 (local address 127.22.26.194:35463)
0927 14:00:34.879640 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:34.879717 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:34.882203 (+ 2486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:34.882984 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:34.882996 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:34.883860 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:34.884755 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:34.884784 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:34.885854 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:34.885861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:34.888033 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:34.888050 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:34.888906 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:34.888916 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:34.889266 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:34.890016 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:00:34.890051 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:34.892761 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:34.895962 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:34.895972 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:34.895990 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:34.896430 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:34.896811 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:34.896816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:34.896819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:34.897010 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:34.897552 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:34.897561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:34.897934 (+ 373us) client_negotiation.cc:769] Sending connection context
0927 14:00:34.898202 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 14:00:34.898795 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":344,"threads_started":1}
I20260927 14:00:34.901631 7659 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39411
I20260927 14:00:34.901986 7659 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:34.902884 7659 heartbeater.cc:511] Master 127.22.26.254:39411 requested a full tablet report, sending...
I20260927 14:00:34.904763 7336 ts_manager.cc:194] Registered new tserver with Master: e775f523ab424156b457c3866f485d6c (127.22.26.194:40737)
I20260927 14:00:34.905668 7336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:35463
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 14:00:34.943677 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39411
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:35.277402 7664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:35.277714 7664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:35.277812 7664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:35.285391 7664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:35.285530 7664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:35.285650 7664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 14:00:35.296242 7664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:39411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:35.297654 7664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:35.299031 7664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:35.313308 7672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:35.313318 7670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:35.313426 7669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:35.314008 7664 server_base.cc:1034] running on GCE node
I20260927 14:00:35.314639 7664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:35.315716 7664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:35.316902 7664 hybrid_clock.cc:648] HybridClock initialized: now 1790517635316830 us; error 61 us; skew 500 ppm
I20260927 14:00:35.317245 7664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:35 dist-test-slave-gkks krb5kdc[7291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517635, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:35.320761 7664 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 14:00:35.322358 7664 webserver.cc:460] Webserver started at http://127.22.26.195:42871/ using document root <none> and password file <none>
I20260927 14:00:35.323194 7664 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:35.323355 7664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:35.323701 7664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:35.327360 7664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "c0602adb2fca4a698ae1f38eb759256e"
format_stamp: "Formatted at 2026-09-27 14:00:35 on dist-test-slave-gkks"
server_key: "4c414635b256e8194ffba94744dd7c2e"
server_key_iv: "d896f4712d503607df085f8a691cff0a"
server_key_version: "encryptionkey@0"
I20260927 14:00:35.328178 7664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "c0602adb2fca4a698ae1f38eb759256e"
format_stamp: "Formatted at 2026-09-27 14:00:35 on dist-test-slave-gkks"
server_key: "4c414635b256e8194ffba94744dd7c2e"
server_key_iv: "d896f4712d503607df085f8a691cff0a"
server_key_version: "encryptionkey@0"
I20260927 14:00:35.335137 7664 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260927 14:00:35.339046 7679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:35.340476 7664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:00:35.340703 7664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "c0602adb2fca4a698ae1f38eb759256e"
format_stamp: "Formatted at 2026-09-27 14:00:35 on dist-test-slave-gkks"
server_key: "4c414635b256e8194ffba94744dd7c2e"
server_key_iv: "d896f4712d503607df085f8a691cff0a"
server_key_version: "encryptionkey@0"
I20260927 14:00:35.340916 7664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:35.383320 7664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:35.387578 7664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:35.387902 7664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:35.388953 7664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:35.390321 7664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:35.390429 7664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:35.390578 7664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:35.390655 7664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:35.402102 7525 heartbeater.cc:503] Master 127.22.26.254:39411 was elected leader, sending a full tablet report...
I20260927 14:00:35.436836 7664 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:42223
I20260927 14:00:35.436899 7791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:42223 every 8 connection(s)
I20260927 14:00:35.438467 7664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 14:00:35.441720 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 7664
I20260927 14:00:35.441926 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 14:00:35.442430 22635 external_mini_cluster.cc:1428] Setting key 666b6c1f987cc23365d1836d6ef75604
Sep 27 14:00:35 dist-test-slave-gkks krb5kdc[7291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517635, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:00:35.463300 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.441650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:57715 (local address 127.22.26.254:39411)
0927 14:00:35.447470 (+ 5820us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:35.447474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:35.447490 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:00:35.447534 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:35.447538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:35.447604 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:35.447692 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:35.449357 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.450173 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.451815 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.452216 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.452315 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.456067 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:35.456074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:35.456077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:35.456112 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:35.459859 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:35.460693 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:35.460777 (+ 84us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:35.460784 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:35.460945 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:35.461411 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:35.461417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:35.461420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:35.461739 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:35.461910 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:35.462661 (+ 751us) server_negotiation.cc:299] Negotiation successful
0927 14:00:35.462882 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5714,"thread_start_us":5584,"threads_started":1}
I20260927 14:00:35.464299 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.442406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:39411 (local address 127.22.26.195:57715)
0927 14:00:35.443750 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:35.443809 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:35.446039 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:35.447927 (+ 1888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:35.447935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:35.448507 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:35.449178 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:35.449198 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.450313 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.450317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:35.451680 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:35.451696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.452400 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.452404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:35.452599 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.453400 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:00:35.453437 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:35.455805 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:35.460025 (+ 4220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:35.460035 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:35.460053 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:35.460538 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:35.461077 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:35.461082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:35.461086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:35.461287 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:35.461890 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:35.461899 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:35.462244 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 14:00:35.462504 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 14:00:35.463072 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":741,"threads_started":1}
I20260927 14:00:35.465755 7792 heartbeater.cc:347] Connected to a master server at 127.22.26.254:39411
I20260927 14:00:35.466094 7792 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:35.466786 7792 heartbeater.cc:511] Master 127.22.26.254:39411 requested a full tablet report, sending...
I20260927 14:00:35.468338 7336 ts_manager.cc:194] Registered new tserver with Master: c0602adb2fca4a698ae1f38eb759256e (127.22.26.195:42223)
I20260927 14:00:35.469201 7336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:57715
I20260927 14:00:35.471683 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:00:35.488504 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.476265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.22.26.254:39411)
0927 14:00:35.476487 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:35.476494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:35.476749 (+ 255us) server_negotiation.cc:407] Connection header received
0927 14:00:35.476870 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:35.476874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:35.476932 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:35.477010 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:35.478405 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.479233 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.480512 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.480961 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.481062 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.483478 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:35.483483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:35.483486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:35.483516 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:35.485945 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:35.486651 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:35.486655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:35.486657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:35.486772 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:35.487159 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:35.487163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:35.487165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:35.487480 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:35.487716 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:35.488001 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 14:00:35.488169 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 14:00:35.492455 7336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:00:35.499327 7336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:00:35.532024 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.511787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.22.26.195:42223)
0927 14:00:35.514119 (+ 2332us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:35.514131 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:35.516447 (+ 2316us) server_negotiation.cc:407] Connection header received
0927 14:00:35.516943 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:35.516957 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:35.517327 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:35.517527 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:35.520272 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.522687 (+ 2415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.528362 (+ 5675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.530269 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.530381 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.530658 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:35.531363 (+ 705us) server_negotiation.cc:299] Negotiation successful
0927 14:00:35.531581 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2096,"thread_start_us":1886,"threads_started":1}
I20260927 14:00:35.532938 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.511554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:42223 (local address 127.0.0.1:48226)
0927 14:00:35.514863 (+ 3309us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:35.516321 (+ 1458us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:35.516646 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:35.517994 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:35.518022 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:35.518067 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:35.518624 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:35.518642 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.522940 (+ 4298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.522950 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:35.528005 (+ 5055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:35.528030 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.530633 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.530641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:35.530991 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.531098 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 14:00:35.532279 (+ 1181us) client_negotiation.cc:240] Negotiation successful
0927 14:00:35.532495 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2960,"mutex_wait_us":65,"thread_start_us":17601,"threads_started":1}
I20260927 14:00:35.537781 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.511592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:40737 (local address 127.0.0.1:52492)
0927 14:00:35.528851 (+ 17259us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:35.528876 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:35.529789 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:35.530791 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:35.530796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:35.530825 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:35.531462 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:35.531479 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.533592 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.533597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:35.535298 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:35.535314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.536959 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.536966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:35.537126 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.537165 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 14:00:35.537266 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 14:00:35.537393 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17085,"mutex_wait_us":64,"thread_start_us":2729,"threads_started":1}
I20260927 14:00:35.537822 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.512240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.22.26.194:40737)
0927 14:00:35.513472 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:35.513487 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:35.529001 (+ 15514us) server_negotiation.cc:407] Connection header received
0927 14:00:35.530024 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:35.530036 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:35.530338 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:35.530514 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:35.531707 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.533446 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.535481 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.536759 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.536912 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.537105 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:35.537323 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 14:00:35.537486 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":991,"thread_start_us":794,"threads_started":1}
I20260927 14:00:35.537396 7727 tablet_service.cc:1467] Processing CreateTablet for tablet 63c16c18a4e24748901968f33fc738eb (DEFAULT_TABLE table=test-table [id=89bb91866d6541e5be99517dbbd31c7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:35.538877 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.515011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:33073 (local address 127.0.0.1:43870)
0927 14:00:35.529448 (+ 14437us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:35.529468 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:35.529606 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:35.531075 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:35.531100 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:35.531127 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:35.531563 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:35.531572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.533166 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.533171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:35.535663 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:35.535679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.537958 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.537965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:35.538170 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.538209 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 14:00:35.538298 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 14:00:35.538441 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14294,"mutex_wait_us":138,"thread_start_us":13241,"threads_started":1}
I20260927 14:00:35.539278 7727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c16c18a4e24748901968f33fc738eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:35.539819 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.515431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.22.26.193:33073)
0927 14:00:35.530522 (+ 15091us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:35.530530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:35.530555 (+ 25us) server_negotiation.cc:407] Connection header received
0927 14:00:35.530654 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:35.530661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:35.530880 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:35.531087 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:35.531723 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.532969 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.535867 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.537380 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.537560 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.537793 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:35.539263 (+ 1470us) server_negotiation.cc:299] Negotiation successful
0927 14:00:35.539447 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14902,"thread_start_us":15272,"threads_started":1}
I20260927 14:00:35.541429 7594 tablet_service.cc:1467] Processing CreateTablet for tablet 63c16c18a4e24748901968f33fc738eb (DEFAULT_TABLE table=test-table [id=89bb91866d6541e5be99517dbbd31c7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:35.542862 7594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c16c18a4e24748901968f33fc738eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:35.543598 7460 tablet_service.cc:1467] Processing CreateTablet for tablet 63c16c18a4e24748901968f33fc738eb (DEFAULT_TABLE table=test-table [id=89bb91866d6541e5be99517dbbd31c7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:35.546236 7460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c16c18a4e24748901968f33fc738eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:35.553521 7811 tablet_bootstrap.cc:492] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e: Bootstrap starting.
I20260927 14:00:35.557088 7811 tablet_bootstrap.cc:654] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:35.558177 7811 log.cc:824] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:35.560086 7812 tablet_bootstrap.cc:492] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c: Bootstrap starting.
I20260927 14:00:35.561769 7814 tablet_bootstrap.cc:492] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86: Bootstrap starting.
I20260927 14:00:35.563077 7811 tablet_bootstrap.cc:492] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e: No bootstrap required, opened a new log
I20260927 14:00:35.563401 7811 ts_tablet_manager.cc:1397] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260927 14:00:35.564520 7812 tablet_bootstrap.cc:654] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:35.566177 7812 log.cc:824] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:35.566599 7814 tablet_bootstrap.cc:654] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:35.567973 7814 log.cc:824] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:35.569602 7812 tablet_bootstrap.cc:492] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c: No bootstrap required, opened a new log
I20260927 14:00:35.569903 7812 ts_tablet_manager.cc:1397] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260927 14:00:35.571280 7814 tablet_bootstrap.cc:492] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86: No bootstrap required, opened a new log
I20260927 14:00:35.571614 7814 ts_tablet_manager.cc:1397] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 14:00:35.572475 7811 raft_consensus.cc:348] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0602adb2fca4a698ae1f38eb759256e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42223 } } peers { permanent_uuid: "e775f523ab424156b457c3866f485d6c" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40737 } } peers { permanent_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33073 } }
I20260927 14:00:35.572922 7811 raft_consensus.cc:374] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:35.573021 7811 raft_consensus.cc:729] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0602adb2fca4a698ae1f38eb759256e, State: Initialized, Role: FOLLOWER
I20260927 14:00:35.573653 7811 consensus_queue.cc:260] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0602adb2fca4a698ae1f38eb759256e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42223 } } peers { permanent_uuid: "e775f523ab424156b457c3866f485d6c" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40737 } } peers { permanent_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33073 } }
I20260927 14:00:35.574800 7792 heartbeater.cc:503] Master 127.22.26.254:39411 was elected leader, sending a full tablet report...
I20260927 14:00:35.575543 7811 ts_tablet_manager.cc:1428] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260927 14:00:35.582598 7812 raft_consensus.cc:348] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0602adb2fca4a698ae1f38eb759256e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42223 } } peers { permanent_uuid: "e775f523ab424156b457c3866f485d6c" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40737 } } peers { permanent_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33073 } }
I20260927 14:00:35.582999 7812 raft_consensus.cc:374] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:35.583122 7812 raft_consensus.cc:729] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e775f523ab424156b457c3866f485d6c, State: Initialized, Role: FOLLOWER
I20260927 14:00:35.582865 7814 raft_consensus.cc:348] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0602adb2fca4a698ae1f38eb759256e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42223 } } peers { permanent_uuid: "e775f523ab424156b457c3866f485d6c" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40737 } } peers { permanent_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33073 } }
I20260927 14:00:35.583412 7814 raft_consensus.cc:374] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:35.583546 7814 raft_consensus.cc:729] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e6a2ac4aea4f53a70d43bdad1b1c86, State: Initialized, Role: FOLLOWER
I20260927 14:00:35.583719 7812 consensus_queue.cc:260] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0602adb2fca4a698ae1f38eb759256e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42223 } } peers { permanent_uuid: "e775f523ab424156b457c3866f485d6c" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40737 } } peers { permanent_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33073 } }
I20260927 14:00:35.584177 7814 consensus_queue.cc:260] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0602adb2fca4a698ae1f38eb759256e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42223 } } peers { permanent_uuid: "e775f523ab424156b457c3866f485d6c" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40737 } } peers { permanent_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33073 } }
I20260927 14:00:35.584913 7659 heartbeater.cc:503] Master 127.22.26.254:39411 was elected leader, sending a full tablet report...
I20260927 14:00:35.585549 7812 ts_tablet_manager.cc:1428] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c: Time spent starting tablet: real 0.016s user 0.008s sys 0.006s
I20260927 14:00:35.586010 7814 ts_tablet_manager.cc:1428] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260927 14:00:35.618425 7526 tablet.cc:2369] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:00:35.627624 7660 tablet.cc:2369] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:00:35.686949 7819 raft_consensus.cc:482] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:00:35.687279 7819 raft_consensus.cc:504] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0602adb2fca4a698ae1f38eb759256e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42223 } } peers { permanent_uuid: "e775f523ab424156b457c3866f485d6c" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40737 } } peers { permanent_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33073 } }
I20260927 14:00:35.689149 7819 leader_election.cc:288] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0602adb2fca4a698ae1f38eb759256e (127.22.26.195:42223), e775f523ab424156b457c3866f485d6c (127.22.26.194:40737)
W20260927 14:00:35.690412 7793 tablet.cc:2369] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:00:35.698726 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.689685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:35213 (local address 127.22.26.194:40737)
0927 14:00:35.689898 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:35.689906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:35.691072 (+ 1166us) server_negotiation.cc:407] Connection header received
0927 14:00:35.691271 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:35.691276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:35.691363 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:35.691468 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:35.692121 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.693168 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.696320 (+ 3152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.697564 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.697733 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.697808 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:35.698314 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 14:00:35.698449 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 14:00:35.698726 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.689650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:37345 (local address 127.22.26.195:42223)
0927 14:00:35.689841 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:35.689849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:35.690628 (+ 779us) server_negotiation.cc:407] Connection header received
0927 14:00:35.691031 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:35.691037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:35.691144 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:35.691235 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:35.691924 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.693094 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.696319 (+ 3225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.697564 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.697733 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.697808 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:35.698314 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 14:00:35.698447 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:00:35.698951 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.689633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:42223 (local address 127.22.26.193:37345)
0927 14:00:35.690465 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:35.690488 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:35.690837 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:35.691272 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:35.691276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:35.691307 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:35.691755 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:35.691768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.693282 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.693290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:35.696118 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:35.696126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.697868 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.697878 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:35.698103 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.698156 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 14:00:35.698266 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 14:00:35.698429 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":286,"thread_start_us":418,"threads_started":1}
I20260927 14:00:35.698985 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.690170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:40737 (local address 127.22.26.193:35213)
0927 14:00:35.690947 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:35.690966 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:35.691123 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:35.691507 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:35.691517 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:35.691540 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:35.691971 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:35.691980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.693297 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.693300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:35.696108 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:35.696124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.697892 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.697898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:35.698103 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.698156 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 14:00:35.698266 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 14:00:35.698428 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":209,"thread_start_us":414,"threads_started":1}
I20260927 14:00:35.700173 7747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c16c18a4e24748901968f33fc738eb" candidate_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0602adb2fca4a698ae1f38eb759256e" is_pre_election: true
I20260927 14:00:35.700176 7614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c16c18a4e24748901968f33fc738eb" candidate_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775f523ab424156b457c3866f485d6c" is_pre_election: true
I20260927 14:00:35.700655 7614 raft_consensus.cc:2454] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55e6a2ac4aea4f53a70d43bdad1b1c86 in term 0.
I20260927 14:00:35.700654 7747 raft_consensus.cc:2454] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55e6a2ac4aea4f53a70d43bdad1b1c86 in term 0.
I20260927 14:00:35.701351 7415 leader_election.cc:302] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e6a2ac4aea4f53a70d43bdad1b1c86, e775f523ab424156b457c3866f485d6c; no voters:
I20260927 14:00:35.701778 7819 raft_consensus.cc:2781] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:00:35.701932 7819 raft_consensus.cc:482] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:00:35.702040 7819 raft_consensus.cc:3037] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:35.704488 7819 raft_consensus.cc:504] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0602adb2fca4a698ae1f38eb759256e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42223 } } peers { permanent_uuid: "e775f523ab424156b457c3866f485d6c" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40737 } } peers { permanent_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33073 } }
I20260927 14:00:35.705067 7819 leader_election.cc:288] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [CANDIDATE]: Term 1 election: Requested vote from peers c0602adb2fca4a698ae1f38eb759256e (127.22.26.195:42223), e775f523ab424156b457c3866f485d6c (127.22.26.194:40737)
I20260927 14:00:35.705448 7747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c16c18a4e24748901968f33fc738eb" candidate_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0602adb2fca4a698ae1f38eb759256e"
I20260927 14:00:35.705575 7614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c16c18a4e24748901968f33fc738eb" candidate_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775f523ab424156b457c3866f485d6c"
I20260927 14:00:35.705649 7747 raft_consensus.cc:3037] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:35.705806 7614 raft_consensus.cc:3037] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:35.707942 7747 raft_consensus.cc:2454] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e6a2ac4aea4f53a70d43bdad1b1c86 in term 1.
I20260927 14:00:35.708277 7614 raft_consensus.cc:2454] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e6a2ac4aea4f53a70d43bdad1b1c86 in term 1.
I20260927 14:00:35.708415 7414 leader_election.cc:302] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e6a2ac4aea4f53a70d43bdad1b1c86, c0602adb2fca4a698ae1f38eb759256e; no voters:
I20260927 14:00:35.708747 7819 raft_consensus.cc:2781] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:35.709177 7819 raft_consensus.cc:686] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [term 1 LEADER]: Becoming Leader. State: Replica: 55e6a2ac4aea4f53a70d43bdad1b1c86, State: Running, Role: LEADER
I20260927 14:00:35.709658 7819 consensus_queue.cc:237] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0602adb2fca4a698ae1f38eb759256e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42223 } } peers { permanent_uuid: "e775f523ab424156b457c3866f485d6c" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40737 } } peers { permanent_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33073 } }
I20260927 14:00:35.714938 7335 catalog_manager.cc:5495] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55e6a2ac4aea4f53a70d43bdad1b1c86 (127.22.26.193). New cstate: current_term: 1 leader_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0602adb2fca4a698ae1f38eb759256e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775f523ab424156b457c3866f485d6c" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55e6a2ac4aea4f53a70d43bdad1b1c86" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 33073 } health_report { overall_health: HEALTHY } } }
I20260927 14:00:35.762863 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.756989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.22.26.193:33073)
0927 14:00:35.757205 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:35.757213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:35.757235 (+ 22us) server_negotiation.cc:407] Connection header received
0927 14:00:35.757394 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:35.757398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:35.757479 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:35.757641 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:00:35.758002 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.758773 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.760340 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.760795 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.760897 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.761617 (+ 720us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:00:35.762192 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:00:35.762320 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:35.762545 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 14:00:35.762647 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 14:00:35.771414 7614 raft_consensus.cc:1261] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c [term 1 FOLLOWER]: Refusing update from remote peer 55e6a2ac4aea4f53a70d43bdad1b1c86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:00:35.771411 7747 raft_consensus.cc:1261] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e [term 1 FOLLOWER]: Refusing update from remote peer 55e6a2ac4aea4f53a70d43bdad1b1c86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:00:35.772349 7822 consensus_queue.cc:1035] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0602adb2fca4a698ae1f38eb759256e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 42223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:00:35.772758 7819 consensus_queue.cc:1035] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775f523ab424156b457c3866f485d6c" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 40737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:00:35.783185 7826 mvcc.cc:205] Tried to move back new op lower bound from 7333960236109635584 to 7333960235870687232. Current Snapshot: MvccSnapshot[applied={T|T < 7333960236109635584}]
I20260927 14:00:35.798995 7336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:00:35.799350 7336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:00:35.803184 7336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 555c0f7eea434dfc9289c2b978b95ce3: Sending DeleteTablet for 3 replicas of tablet 63c16c18a4e24748901968f33fc738eb
I20260927 14:00:35.804436 7727 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c16c18a4e24748901968f33fc738eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:35 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:48226
I20260927 14:00:35.804755 7594 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c16c18a4e24748901968f33fc738eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:35 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:52492
I20260927 14:00:35.805538 7460 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c16c18a4e24748901968f33fc738eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:35 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:43870
I20260927 14:00:35.805734 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 7397
I20260927 14:00:35.825657 7837 tablet_replica.cc:330] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86: stopping tablet replica
I20260927 14:00:35.826189 7837 raft_consensus.cc:2229] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:00:35.826908 7837 raft_consensus.cc:2258] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:35.829870 7837 ts_tablet_manager.cc:1905] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 14:00:35 dist-test-slave-gkks krb5kdc[7291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 14:00:35.831355 7836 tablet_replica.cc:330] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c: stopping tablet replica
I20260927 14:00:35.831809 7836 raft_consensus.cc:2229] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:00:35.832376 7836 raft_consensus.cc:2258] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:35.841461 7836 ts_tablet_manager.cc:1905] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:35.842265 7837 ts_tablet_manager.cc:1918] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:35.842454 7837 log.cc:1196] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/63c16c18a4e24748901968f33fc738eb
I20260927 14:00:35.843071 7837 ts_tablet_manager.cc:1939] T 63c16c18a4e24748901968f33fc738eb P 55e6a2ac4aea4f53a70d43bdad1b1c86: Deleting consensus metadata
I20260927 14:00:35.845477 7322 catalog_manager.cc:4841] TS 55e6a2ac4aea4f53a70d43bdad1b1c86 (127.22.26.193:33073): tablet 63c16c18a4e24748901968f33fc738eb (table test-table [id=89bb91866d6541e5be99517dbbd31c7c]) successfully deleted
I20260927 14:00:35.846187 7835 tablet_replica.cc:330] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e: stopping tablet replica
I20260927 14:00:35.846628 7835 raft_consensus.cc:2229] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:00:35.847186 7835 raft_consensus.cc:2258] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:35.848449 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.807075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.22.26.193:33073)
0927 14:00:35.807272 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:35.807279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:35.817058 (+ 9779us) server_negotiation.cc:407] Connection header received
0927 14:00:35.817550 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:35.817557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:35.817646 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:35.817735 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:35.819811 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.821053 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.823160 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.823911 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.824012 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.833930 (+ 9918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:35.833946 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:35.833963 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:35.834038 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:35.838588 (+ 4550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:35.839576 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:35.839584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:35.839588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:35.839786 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:35.844758 (+ 4972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:35.844764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:35.844768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:35.845235 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:35.845524 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:35.847709 (+ 2185us) server_negotiation.cc:299] Negotiation successful
0927 14:00:35.847967 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:00:35.849401 7835 ts_tablet_manager.cc:1905] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:35.849835 7836 ts_tablet_manager.cc:1918] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:35.849992 7836 log.cc:1196] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/63c16c18a4e24748901968f33fc738eb
I20260927 14:00:35.850515 7836 ts_tablet_manager.cc:1939] T 63c16c18a4e24748901968f33fc738eb P e775f523ab424156b457c3866f485d6c: Deleting consensus metadata
I20260927 14:00:35.850638 7520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:35.853101 7322 catalog_manager.cc:4841] TS e775f523ab424156b457c3866f485d6c (127.22.26.194:40737): tablet 63c16c18a4e24748901968f33fc738eb (table test-table [id=89bb91866d6541e5be99517dbbd31c7c]) successfully deleted
I20260927 14:00:35.857673 7835 ts_tablet_manager.cc:1918] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:35.857826 7835 log.cc:1196] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/63c16c18a4e24748901968f33fc738eb
I20260927 14:00:35.858296 7835 ts_tablet_manager.cc:1939] T 63c16c18a4e24748901968f33fc738eb P c0602adb2fca4a698ae1f38eb759256e: Deleting consensus metadata
I20260927 14:00:35.859993 7321 catalog_manager.cc:4841] TS c0602adb2fca4a698ae1f38eb759256e (127.22.26.195:42223): tablet 63c16c18a4e24748901968f33fc738eb (table test-table [id=89bb91866d6541e5be99517dbbd31c7c]) successfully deleted
I20260927 14:00:35.953408 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 7397
I20260927 14:00:35.972822 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 7531
Sep 27 14:00:35 dist-test-slave-gkks krb5kdc[7291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 14:00:35.988646 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:35.973606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.22.26.194:40737)
0927 14:00:35.973806 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:35.973812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:35.973911 (+ 99us) server_negotiation.cc:407] Connection header received
0927 14:00:35.974142 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:35.974148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:35.974226 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:35.974331 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:35.975817 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.976677 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.978034 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:35.978430 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:35.978557 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:35.982644 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:35.982667 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:35.982683 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:35.982724 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:35.985450 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:35.986297 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:35.986305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:35.986307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:35.986488 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:35.986965 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:35.986970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:35.986974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:35.987453 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:35.987755 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:35.988062 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 14:00:35.988305 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 14:00:35.989282 7654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:36.078352 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 7531
I20260927 14:00:36.095692 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 7664
Sep 27 14:00:36 dist-test-slave-gkks krb5kdc[7291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 14:00:36.111579 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:36.096492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.22.26.195:42223)
0927 14:00:36.096721 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:36.096727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:36.096748 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:00:36.096955 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:36.096961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:36.097051 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:36.097163 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:36.098567 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:36.099381 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:36.100671 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:36.101076 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:36.101216 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:36.105761 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:36.105776 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:36.105792 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:36.105833 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:36.108620 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:36.109326 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:36.109334 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:36.109336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:36.109490 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:36.109868 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:36.109873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:36.109877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:36.110224 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:36.110516 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:36.110851 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 14:00:36.111153 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 14:00:36.112201 7787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:36.201956 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 7664
I20260927 14:00:36.218575 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 7304
I20260927 14:00:36.219867 7366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:36.332613 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 7304
2026-09-27T14:00:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4481 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T14:00:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:00:36Z Disabled control of system clock
I20260927 14:00:36.414316 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:41527
--webserver_interface=127.22.26.254
--webserver_port=0
--builtin_ntp_servers=127.22.26.212:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:41527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 14:00:36.738538 7846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:36.738816 7846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:36.738898 7846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:36.746095 7846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:00:36.746198 7846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:36.746235 7846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:36.746290 7846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:00:36.746333 7846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:00:36.755496 7846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:41527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:41527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:36.756747 7846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:36.758446 7846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:36.769903 7851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:36.769871 7852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:36.769935 7854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:36.771134 7846 server_base.cc:1034] running on GCE node
I20260927 14:00:36.771915 7846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:36.773393 7846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:36.774629 7846 hybrid_clock.cc:648] HybridClock initialized: now 1790517636774590 us; error 49 us; skew 500 ppm
I20260927 14:00:36.774978 7846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 14:00:36.781700 7846 webserver.cc:460] Webserver started at http://127.22.26.254:44181/ using document root <none> and password file <none>
I20260927 14:00:36.782425 7846 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:36.782562 7846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:36.782891 7846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:36.785817 7846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "b638e7b695a74eb282b1bd707233dab1"
format_stamp: "Formatted at 2026-09-27 14:00:36 on dist-test-slave-gkks"
server_key: "2d11c564f90e005222ddab575feed691"
server_key_iv: "fd10370d69bb4b8e0c60e8f0679bb8e2"
server_key_version: "encryptionkey@0"
I20260927 14:00:36.786616 7846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "b638e7b695a74eb282b1bd707233dab1"
format_stamp: "Formatted at 2026-09-27 14:00:36 on dist-test-slave-gkks"
server_key: "2d11c564f90e005222ddab575feed691"
server_key_iv: "fd10370d69bb4b8e0c60e8f0679bb8e2"
server_key_version: "encryptionkey@0"
I20260927 14:00:36.791922 7846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 14:00:36.795760 7860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:36.797175 7846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:00:36.797374 7846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "b638e7b695a74eb282b1bd707233dab1"
format_stamp: "Formatted at 2026-09-27 14:00:36 on dist-test-slave-gkks"
server_key: "2d11c564f90e005222ddab575feed691"
server_key_iv: "fd10370d69bb4b8e0c60e8f0679bb8e2"
server_key_version: "encryptionkey@0"
I20260927 14:00:36.797600 7846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:36.823307 7846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:36.824156 7846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:36.824441 7846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:36.842626 7846 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:41527
I20260927 14:00:36.842713 7911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:41527 every 8 connection(s)
I20260927 14:00:36.843977 7846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:00:36.846685 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 7846
I20260927 14:00:36.846889 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:00:36.847610 22635 external_mini_cluster.cc:1428] Setting key 073bef4ed3242a7808f7817d75c4fcbb
I20260927 14:00:36.848392 7912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:36.861583 7912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1: Bootstrap starting.
I20260927 14:00:36.862082 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:36.849715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.22.26.254:41527)
0927 14:00:36.850723 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:36.850734 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:36.850773 (+ 39us) server_negotiation.cc:407] Connection header received
0927 14:00:36.853202 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:36.853224 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:36.853702 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:36.854114 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:36.854702 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:36.856110 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:36.857531 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:36.858200 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:36.858425 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:36.859749 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:36.859758 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:36.859771 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:36.860319 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:36.860395 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:36.860419 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:36.860549 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:36.860779 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 14:00:36.861079 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":390,"threads_started":1}
I20260927 14:00:36.864861 7912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:36.865854 7912 log.cc:824] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:36.868631 7912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1: No bootstrap required, opened a new log
I20260927 14:00:36.876537 7912 raft_consensus.cc:348] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b638e7b695a74eb282b1bd707233dab1" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 41527 } }
I20260927 14:00:36.876861 7912 raft_consensus.cc:374] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:36.876951 7912 raft_consensus.cc:729] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b638e7b695a74eb282b1bd707233dab1, State: Initialized, Role: FOLLOWER
I20260927 14:00:36.877475 7912 consensus_queue.cc:260] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b638e7b695a74eb282b1bd707233dab1" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 41527 } }
I20260927 14:00:36.877696 7912 raft_consensus.cc:388] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:00:36.877827 7912 raft_consensus.cc:482] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:00:36.878015 7912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:36.880548 7912 raft_consensus.cc:504] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b638e7b695a74eb282b1bd707233dab1" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 41527 } }
I20260927 14:00:36.881098 7912 leader_election.cc:302] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b638e7b695a74eb282b1bd707233dab1; no voters:
I20260927 14:00:36.881596 7912 leader_election.cc:288] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:00:36.881769 7917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:36.882530 7917 raft_consensus.cc:686] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [term 1 LEADER]: Becoming Leader. State: Replica: b638e7b695a74eb282b1bd707233dab1, State: Running, Role: LEADER
I20260927 14:00:36.883062 7917 consensus_queue.cc:237] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b638e7b695a74eb282b1bd707233dab1" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 41527 } }
I20260927 14:00:36.883713 7912 sys_catalog.cc:564] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:00:36.885865 7919 sys_catalog.cc:455] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b638e7b695a74eb282b1bd707233dab1. Latest consensus state: current_term: 1 leader_uuid: "b638e7b695a74eb282b1bd707233dab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b638e7b695a74eb282b1bd707233dab1" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 41527 } } }
I20260927 14:00:36.886282 7919 sys_catalog.cc:458] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:36.886284 7918 sys_catalog.cc:455] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b638e7b695a74eb282b1bd707233dab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b638e7b695a74eb282b1bd707233dab1" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 41527 } } }
I20260927 14:00:36.886648 7918 sys_catalog.cc:458] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:36.888371 7923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:00:36.892676 7923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:00:36.902663 7923 catalog_manager.cc:1347] Generated new cluster ID: bdf28498369e46a9bae9311c5b380165
I20260927 14:00:36.902797 7923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:00:36.918856 7923 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:00:36.920320 7923 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:00:36.932737 7923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1: Generated new TSK 0
I20260927 14:00:36.933605 7923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:00:36.941817 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:41527
--builtin_ntp_servers=127.22.26.212:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 14:00:37.270946 7936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:37.271294 7936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:37.271421 7936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:37.278718 7936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:37.278810 7936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:37.278968 7936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 14:00:37.288359 7936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:41527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:37.289707 7936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:37.291499 7936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:37.305094 7941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:37.305068 7942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:37.305182 7944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:37.305759 7936 server_base.cc:1034] running on GCE node
I20260927 14:00:37.306413 7936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:37.307442 7936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:37.308663 7936 hybrid_clock.cc:648] HybridClock initialized: now 1790517637308620 us; error 47 us; skew 500 ppm
I20260927 14:00:37.308979 7936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 14:00:37.311395 7936 webserver.cc:460] Webserver started at http://127.22.26.193:34355/ using document root <none> and password file <none>
I20260927 14:00:37.312093 7936 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:37.312224 7936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:37.312539 7936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:37.315507 7936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "c02df5024e4146268712cfc294dc0b00"
format_stamp: "Formatted at 2026-09-27 14:00:37 on dist-test-slave-gkks"
server_key: "b8490f975f94702edd745308693f0958"
server_key_iv: "d3243736bdc8186ca9c039d7a60f2916"
server_key_version: "encryptionkey@0"
I20260927 14:00:37.316313 7936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "c02df5024e4146268712cfc294dc0b00"
format_stamp: "Formatted at 2026-09-27 14:00:37 on dist-test-slave-gkks"
server_key: "b8490f975f94702edd745308693f0958"
server_key_iv: "d3243736bdc8186ca9c039d7a60f2916"
server_key_version: "encryptionkey@0"
I20260927 14:00:37.321465 7936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260927 14:00:37.325220 7950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:37.326520 7936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 14:00:37.326710 7936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "c02df5024e4146268712cfc294dc0b00"
format_stamp: "Formatted at 2026-09-27 14:00:37 on dist-test-slave-gkks"
server_key: "b8490f975f94702edd745308693f0958"
server_key_iv: "d3243736bdc8186ca9c039d7a60f2916"
server_key_version: "encryptionkey@0"
I20260927 14:00:37.326947 7936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:37.355960 7936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:37.356918 7936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:37.357268 7936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:37.358307 7936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:37.359941 7936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:37.360096 7936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:37.360252 7936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:37.360361 7936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:37.394742 7936 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:35311
I20260927 14:00:37.394829 8062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:35311 every 8 connection(s)
I20260927 14:00:37.396201 7936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 14:00:37.405783 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 7936
I20260927 14:00:37.405980 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 14:00:37.406574 22635 external_mini_cluster.cc:1428] Setting key 926325bd75be5a04f75e792243152372
I20260927 14:00:37.409436 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:41527
--builtin_ntp_servers=127.22.26.212:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 14:00:37.409600 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:37.398841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:54499 (local address 127.22.26.254:41527)
0927 14:00:37.399304 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:37.399308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:37.400410 (+ 1102us) server_negotiation.cc:407] Connection header received
0927 14:00:37.402277 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:37.402283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:37.402401 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:37.402525 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:37.403650 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:37.404369 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:37.406335 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:37.406826 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:37.406931 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:37.408625 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:37.408631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:37.408634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:37.408654 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:37.408697 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:37.408710 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:37.408821 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:37.409089 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 14:00:37.409195 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":261,"threads_started":1}
I20260927 14:00:37.410655 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:37.399359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:41527 (local address 127.22.26.193:54499)
0927 14:00:37.400251 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:37.400296 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:37.401997 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:37.402765 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:37.402773 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:37.402861 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:37.403473 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:37.403490 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:37.404528 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:37.404533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:37.406124 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:37.406140 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:37.407053 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:37.407062 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:37.407348 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:37.408196 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:00:37.408200 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:37.408294 (+ 94us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:00:37.408406 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:37.408820 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:37.408833 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:37.408844 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 14:00:37.409067 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 14:00:37.409457 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":292,"threads_started":1}
I20260927 14:00:37.412310 8063 heartbeater.cc:347] Connected to a master server at 127.22.26.254:41527
I20260927 14:00:37.412616 8063 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:37.413475 8063 heartbeater.cc:511] Master 127.22.26.254:41527 requested a full tablet report, sending...
I20260927 14:00:37.415896 7877 ts_manager.cc:194] Registered new tserver with Master: c02df5024e4146268712cfc294dc0b00 (127.22.26.193:35311)
I20260927 14:00:37.418080 7877 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.26.193:54499
W20260927 14:00:37.740952 8067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:37.741235 8067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:37.741374 8067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:37.748721 8067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:37.748826 8067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:37.748999 8067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 14:00:37.758175 8067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:41527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:37.759600 8067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:37.761425 8067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:37.774755 8072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:37.774813 8073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:37.774786 8075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:37.775708 8067 server_base.cc:1034] running on GCE node
I20260927 14:00:37.776342 8067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:37.777386 8067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:37.778631 8067 hybrid_clock.cc:648] HybridClock initialized: now 1790517637778572 us; error 57 us; skew 500 ppm
I20260927 14:00:37.778957 8067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 14:00:37.781379 8067 webserver.cc:460] Webserver started at http://127.22.26.194:42399/ using document root <none> and password file <none>
I20260927 14:00:37.782123 8067 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:37.782261 8067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:37.782590 8067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:37.785755 8067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "ddee7737520c416f8fef17e11a456768"
format_stamp: "Formatted at 2026-09-27 14:00:37 on dist-test-slave-gkks"
server_key: "fa759ca8040d6a176dd9b0c1d9ae3732"
server_key_iv: "be1eab8b33da291b160e8189e517f7dd"
server_key_version: "encryptionkey@0"
I20260927 14:00:37.786566 8067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "ddee7737520c416f8fef17e11a456768"
format_stamp: "Formatted at 2026-09-27 14:00:37 on dist-test-slave-gkks"
server_key: "fa759ca8040d6a176dd9b0c1d9ae3732"
server_key_iv: "be1eab8b33da291b160e8189e517f7dd"
server_key_version: "encryptionkey@0"
I20260927 14:00:37.791824 8067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 14:00:37.795382 8081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:37.796792 8067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:00:37.796988 8067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "ddee7737520c416f8fef17e11a456768"
format_stamp: "Formatted at 2026-09-27 14:00:37 on dist-test-slave-gkks"
server_key: "fa759ca8040d6a176dd9b0c1d9ae3732"
server_key_iv: "be1eab8b33da291b160e8189e517f7dd"
server_key_version: "encryptionkey@0"
I20260927 14:00:37.797224 8067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:37.819880 8067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:37.820837 8067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:37.821199 8067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:37.822221 8067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:37.823755 8067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:37.823885 8067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:37.824023 8067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:37.824126 8067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:37.863652 8067 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:45121
I20260927 14:00:37.863689 8193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:45121 every 8 connection(s)
I20260927 14:00:37.865211 8067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 14:00:37.869513 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 8067
I20260927 14:00:37.869712 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 14:00:37.870168 22635 external_mini_cluster.cc:1428] Setting key d05fb6822e27403d47f39aebf3841d18
I20260927 14:00:37.873083 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:41527
--builtin_ntp_servers=127.22.26.212:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 14:00:37.879190 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:37.868082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:33279 (local address 127.22.26.254:41527)
0927 14:00:37.868300 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:37.868307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:37.869764 (+ 1457us) server_negotiation.cc:407] Connection header received
0927 14:00:37.872019 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:37.872025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:37.872099 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:37.872200 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:37.873226 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:37.874439 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:37.876195 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:37.876868 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:37.877007 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:37.878286 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:37.878294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:37.878303 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:37.878329 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:37.878371 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:37.878388 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:37.878481 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:37.878678 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 14:00:37.878787 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 14:00:37.880172 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:37.868542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:41527 (local address 127.22.26.194:33279)
0927 14:00:37.869595 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:37.869641 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:37.871676 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:37.872416 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:37.872425 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:37.872518 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:37.873059 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:37.873076 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:37.874630 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:37.874639 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:37.876059 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:37.876071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:37.877063 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:37.877067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:37.877268 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:37.877971 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:00:37.877973 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:37.878011 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:00:37.878095 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:37.878484 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:37.878491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:37.878497 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 14:00:37.878629 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 14:00:37.878931 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":335,"threads_started":1}
I20260927 14:00:37.881969 8194 heartbeater.cc:347] Connected to a master server at 127.22.26.254:41527
I20260927 14:00:37.882270 8194 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:37.882963 8194 heartbeater.cc:511] Master 127.22.26.254:41527 requested a full tablet report, sending...
I20260927 14:00:37.884460 7877 ts_manager.cc:194] Registered new tserver with Master: ddee7737520c416f8fef17e11a456768 (127.22.26.194:45121)
I20260927 14:00:37.885349 7877 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.26.194:33279
W20260927 14:00:38.203716 8197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:38.204053 8197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:38.204223 8197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:38.211376 8197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:38.211508 8197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:38.211692 8197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 14:00:38.220547 8197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:41527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:38.221917 8197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:38.223734 8197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:38.237033 8203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:38.237097 8202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:38.237027 8205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:38.237767 8197 server_base.cc:1034] running on GCE node
I20260927 14:00:38.238479 8197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:38.239557 8197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:38.240799 8197 hybrid_clock.cc:648] HybridClock initialized: now 1790517638240767 us; error 51 us; skew 500 ppm
I20260927 14:00:38.241156 8197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 14:00:38.243888 8197 webserver.cc:460] Webserver started at http://127.22.26.195:38773/ using document root <none> and password file <none>
I20260927 14:00:38.244645 8197 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:38.244776 8197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:38.245082 8197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:38.248032 8197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "96cc5657efad4dc68a4387d6fae7634e"
format_stamp: "Formatted at 2026-09-27 14:00:38 on dist-test-slave-gkks"
server_key: "170dedc34d4eba0df54b428b3b4bd548"
server_key_iv: "ee72e8e35adc102130e7ebb05a6e969c"
server_key_version: "encryptionkey@0"
I20260927 14:00:38.248844 8197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "96cc5657efad4dc68a4387d6fae7634e"
format_stamp: "Formatted at 2026-09-27 14:00:38 on dist-test-slave-gkks"
server_key: "170dedc34d4eba0df54b428b3b4bd548"
server_key_iv: "ee72e8e35adc102130e7ebb05a6e969c"
server_key_version: "encryptionkey@0"
I20260927 14:00:38.254194 8197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 14:00:38.257800 8211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:38.259577 8197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:00:38.259776 8197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "96cc5657efad4dc68a4387d6fae7634e"
format_stamp: "Formatted at 2026-09-27 14:00:38 on dist-test-slave-gkks"
server_key: "170dedc34d4eba0df54b428b3b4bd548"
server_key_iv: "ee72e8e35adc102130e7ebb05a6e969c"
server_key_version: "encryptionkey@0"
I20260927 14:00:38.260133 8197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:38.276847 8197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:38.277649 8197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:38.277945 8197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:38.278898 8197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:38.280449 8197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:38.280586 8197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:38.280719 8197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:38.280809 8197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:38.317214 8197 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:32787
I20260927 14:00:38.317307 8323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:32787 every 8 connection(s)
I20260927 14:00:38.318625 8197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 14:00:38.320008 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 8197
I20260927 14:00:38.320207 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 14:00:38.320793 22635 external_mini_cluster.cc:1428] Setting key 3d27c7e967649027df6168a11161ff62
I20260927 14:00:38.331943 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.321731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:53211 (local address 127.22.26.254:41527)
0927 14:00:38.321949 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:38.321953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:38.323713 (+ 1760us) server_negotiation.cc:407] Connection header received
0927 14:00:38.325532 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:38.325537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:38.325624 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:38.325753 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:38.326844 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.327674 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.329274 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.329733 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.329840 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.331224 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:38.331229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:38.331233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:38.331250 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:38.331283 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:38.331295 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:38.331401 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:38.331605 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 14:00:38.331714 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 14:00:38.332947 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.322441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:41527 (local address 127.22.26.195:53211)
0927 14:00:38.323560 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:38.323613 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:38.325270 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:38.325990 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:38.325999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:38.326090 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:38.326677 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:38.326695 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.327828 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.327833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:38.329083 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:38.329095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.329937 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.329941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:38.330138 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.330851 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:00:38.330854 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:38.330892 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:00:38.330965 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:38.331401 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:38.331411 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:38.331426 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 14:00:38.331572 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 14:00:38.331864 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":360,"threads_started":1}
I20260927 14:00:38.334364 8324 heartbeater.cc:347] Connected to a master server at 127.22.26.254:41527
I20260927 14:00:38.334654 8324 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:38.335570 8324 heartbeater.cc:511] Master 127.22.26.254:41527 requested a full tablet report, sending...
I20260927 14:00:38.337142 7877 ts_manager.cc:194] Registered new tserver with Master: 96cc5657efad4dc68a4387d6fae7634e (127.22.26.195:32787)
I20260927 14:00:38.338169 7877 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.26.195:53211
I20260927 14:00:38.349085 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:00:38.359313 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.353522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.22.26.254:41527)
0927 14:00:38.353724 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:38.353730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:38.354243 (+ 513us) server_negotiation.cc:407] Connection header received
0927 14:00:38.354347 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:38.354351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:38.354427 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:38.354515 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:38.355039 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.355825 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.357167 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.357573 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.357671 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.358664 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:38.358669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:38.358672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:38.358703 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:38.358735 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:38.358747 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:38.358816 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:38.358944 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 14:00:38.359015 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 14:00:38.363252 7877 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:00:38.370308 7877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:00:38.394114 8336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.382915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:32787 (local address 127.0.0.1:57034)
0927 14:00:38.384995 (+ 2080us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:38.385012 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:38.385116 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:38.386253 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:38.386257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:38.386281 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:38.386660 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:38.386671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.388834 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.388839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:38.391201 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:38.391215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.392981 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.392987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:38.393251 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.393325 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 14:00:38.393475 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 14:00:38.393761 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1965,"mutex_wait_us":93,"thread_start_us":306,"threads_started":1}
I20260927 14:00:38.395273 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.382218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35311 (local address 127.0.0.1:43044)
0927 14:00:38.383742 (+ 1524us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:38.383788 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:38.384020 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:38.385663 (+ 1643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:38.385671 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:38.385702 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:38.386098 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:38.386111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.388082 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.388087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:38.392366 (+ 4279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:38.392381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.394224 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.394229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:38.394601 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.394633 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 14:00:38.394760 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 14:00:38.394877 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"mutex_wait_us":1849,"thread_start_us":2537,"threads_started":1}
I20260927 14:00:38.396279 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.382324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.22.26.193:35311)
0927 14:00:38.384678 (+ 2354us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:38.384690 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:38.384726 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:00:38.384838 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:38.384846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:38.385117 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:38.385375 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:38.386280 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.387898 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.392566 (+ 4668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.394002 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.394177 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.394430 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:38.394821 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 14:00:38.395889 (+ 1068us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2204,"thread_start_us":2084,"threads_started":1}
I20260927 14:00:38.399055 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.385142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:45121 (local address 127.0.0.1:56220)
0927 14:00:38.390323 (+ 5181us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:38.390340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:38.390447 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:38.391068 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:38.391101 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:38.391132 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:38.391616 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:38.391626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.393124 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.393129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:38.396779 (+ 3650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:38.396791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.398361 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.398367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:38.398543 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.398575 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 14:00:38.398652 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 14:00:38.398774 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5069,"thread_start_us":4956,"threads_started":1}
I20260927 14:00:38.399197 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.385367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.22.26.194:45121)
0927 14:00:38.389927 (+ 4560us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:38.389937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:38.390564 (+ 627us) server_negotiation.cc:407] Connection header received
0927 14:00:38.390669 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:38.390679 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:38.390881 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:38.391107 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:38.391801 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.392945 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.396959 (+ 4014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.398167 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.398308 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.398555 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:38.398717 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 14:00:38.398868 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4395,"thread_start_us":4315,"threads_started":1}
I20260927 14:00:38.399740 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.383223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.22.26.195:32787)
0927 14:00:38.385680 (+ 2457us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:38.385688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:38.385718 (+ 30us) server_negotiation.cc:407] Connection header received
0927 14:00:38.385820 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:38.385827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:38.386044 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:38.386873 (+ 829us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:38.387155 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.388681 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.391399 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.392809 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.392957 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.393251 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:38.393524 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 14:00:38.393698 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2292,"thread_start_us":2124,"threads_started":1}
I20260927 14:00:38.400595 7998 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe32f8ba3f843cdb9bb56386a609b2b (DEFAULT_TABLE table=test-table [id=73f4d269bbce41579bd83c74f331ebc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:38.403182 7998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe32f8ba3f843cdb9bb56386a609b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:38.403443 8259 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe32f8ba3f843cdb9bb56386a609b2b (DEFAULT_TABLE table=test-table [id=73f4d269bbce41579bd83c74f331ebc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:38.404043 8129 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe32f8ba3f843cdb9bb56386a609b2b (DEFAULT_TABLE table=test-table [id=73f4d269bbce41579bd83c74f331ebc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:38.404829 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe32f8ba3f843cdb9bb56386a609b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:38.405184 8129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe32f8ba3f843cdb9bb56386a609b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:38.416330 8342 tablet_bootstrap.cc:492] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768: Bootstrap starting.
I20260927 14:00:38.418388 8343 tablet_bootstrap.cc:492] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00: Bootstrap starting.
I20260927 14:00:38.420043 8344 tablet_bootstrap.cc:492] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e: Bootstrap starting.
I20260927 14:00:38.420365 8342 tablet_bootstrap.cc:654] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:38.421989 8063 heartbeater.cc:503] Master 127.22.26.254:41527 was elected leader, sending a full tablet report...
I20260927 14:00:38.422415 8343 tablet_bootstrap.cc:654] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:38.422827 8344 tablet_bootstrap.cc:654] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:38.423676 8343 log.cc:824] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:38.424021 8344 log.cc:824] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:38.421715 8342 log.cc:824] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:38.430625 8344 tablet_bootstrap.cc:492] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e: No bootstrap required, opened a new log
I20260927 14:00:38.430687 8343 tablet_bootstrap.cc:492] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00: No bootstrap required, opened a new log
I20260927 14:00:38.430702 8342 tablet_bootstrap.cc:492] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768: No bootstrap required, opened a new log
I20260927 14:00:38.430981 8342 ts_tablet_manager.cc:1397] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20260927 14:00:38.431022 8343 ts_tablet_manager.cc:1397] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260927 14:00:38.431140 8344 ts_tablet_manager.cc:1397] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260927 14:00:38.438550 8344 raft_consensus.cc:348] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02df5024e4146268712cfc294dc0b00" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35311 } } peers { permanent_uuid: "ddee7737520c416f8fef17e11a456768" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 45121 } } peers { permanent_uuid: "96cc5657efad4dc68a4387d6fae7634e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 32787 } }
I20260927 14:00:38.438959 8344 raft_consensus.cc:374] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:38.439106 8344 raft_consensus.cc:729] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cc5657efad4dc68a4387d6fae7634e, State: Initialized, Role: FOLLOWER
I20260927 14:00:38.439810 8344 consensus_queue.cc:260] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02df5024e4146268712cfc294dc0b00" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35311 } } peers { permanent_uuid: "ddee7737520c416f8fef17e11a456768" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 45121 } } peers { permanent_uuid: "96cc5657efad4dc68a4387d6fae7634e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 32787 } }
I20260927 14:00:38.440929 8324 heartbeater.cc:503] Master 127.22.26.254:41527 was elected leader, sending a full tablet report...
I20260927 14:00:38.441944 8343 raft_consensus.cc:348] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02df5024e4146268712cfc294dc0b00" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35311 } } peers { permanent_uuid: "ddee7737520c416f8fef17e11a456768" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 45121 } } peers { permanent_uuid: "96cc5657efad4dc68a4387d6fae7634e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 32787 } }
I20260927 14:00:38.441937 8342 raft_consensus.cc:348] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02df5024e4146268712cfc294dc0b00" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35311 } } peers { permanent_uuid: "ddee7737520c416f8fef17e11a456768" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 45121 } } peers { permanent_uuid: "96cc5657efad4dc68a4387d6fae7634e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 32787 } }
I20260927 14:00:38.442378 8342 raft_consensus.cc:374] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:38.442378 8343 raft_consensus.cc:374] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:38.442472 8342 raft_consensus.cc:729] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddee7737520c416f8fef17e11a456768, State: Initialized, Role: FOLLOWER
I20260927 14:00:38.442472 8343 raft_consensus.cc:729] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c02df5024e4146268712cfc294dc0b00, State: Initialized, Role: FOLLOWER
I20260927 14:00:38.443099 8344 ts_tablet_manager.cc:1428] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260927 14:00:38.443054 8343 consensus_queue.cc:260] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02df5024e4146268712cfc294dc0b00" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35311 } } peers { permanent_uuid: "ddee7737520c416f8fef17e11a456768" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 45121 } } peers { permanent_uuid: "96cc5657efad4dc68a4387d6fae7634e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 32787 } }
I20260927 14:00:38.443054 8342 consensus_queue.cc:260] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02df5024e4146268712cfc294dc0b00" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35311 } } peers { permanent_uuid: "ddee7737520c416f8fef17e11a456768" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 45121 } } peers { permanent_uuid: "96cc5657efad4dc68a4387d6fae7634e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 32787 } }
I20260927 14:00:38.444674 8194 heartbeater.cc:503] Master 127.22.26.254:41527 was elected leader, sending a full tablet report...
I20260927 14:00:38.444940 8343 ts_tablet_manager.cc:1428] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 14:00:38.445261 8342 ts_tablet_manager.cc:1428] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260927 14:00:38.448745 8350 raft_consensus.cc:482] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:00:38.449012 8350 raft_consensus.cc:504] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02df5024e4146268712cfc294dc0b00" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35311 } } peers { permanent_uuid: "ddee7737520c416f8fef17e11a456768" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 45121 } } peers { permanent_uuid: "96cc5657efad4dc68a4387d6fae7634e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 32787 } }
I20260927 14:00:38.450686 8350 leader_election.cc:288] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddee7737520c416f8fef17e11a456768 (127.22.26.194:45121), 96cc5657efad4dc68a4387d6fae7634e (127.22.26.195:32787)
I20260927 14:00:38.459597 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.451123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:40773 (local address 127.22.26.194:45121)
0927 14:00:38.451351 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:38.451362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:38.451944 (+ 582us) server_negotiation.cc:407] Connection header received
0927 14:00:38.452278 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:38.452282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:38.452380 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:38.452475 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:38.453192 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.454283 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.457268 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.458455 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.458616 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.458685 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:38.459228 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 14:00:38.459347 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 14:00:38.459599 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.451558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:35611 (local address 127.22.26.195:32787)
0927 14:00:38.451791 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:38.451799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:38.452206 (+ 407us) server_negotiation.cc:407] Connection header received
0927 14:00:38.452369 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:38.452373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:38.452446 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:38.452530 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:38.453234 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.454283 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.457265 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.458461 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.458620 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.458712 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:38.459214 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 14:00:38.459344 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 14:00:38.459805 8351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.450803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:45121 (local address 127.22.26.193:40773)
0927 14:00:38.451823 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:38.451854 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:38.452059 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:38.452515 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:38.452521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:38.452548 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:38.453018 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:38.453037 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.454429 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.454435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:38.457039 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:38.457050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.458696 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.458702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:38.458917 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.459045 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 14:00:38.459166 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 14:00:38.459388 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":245,"thread_start_us":674,"threads_started":1}
I20260927 14:00:38.460281 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.451195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:32787 (local address 127.22.26.193:35611)
0927 14:00:38.452107 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:38.452125 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:38.452221 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:38.452546 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:38.452550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:38.452569 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:38.453057 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:38.453065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.454430 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.454435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:38.457035 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:38.457050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.458672 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.458685 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:38.458917 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.459041 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 14:00:38.459163 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 14:00:38.459388 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":278,"thread_start_us":627,"threads_started":1}
I20260927 14:00:38.460783 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe32f8ba3f843cdb9bb56386a609b2b" candidate_uuid: "c02df5024e4146268712cfc294dc0b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddee7737520c416f8fef17e11a456768" is_pre_election: true
I20260927 14:00:38.461280 8149 raft_consensus.cc:2454] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c02df5024e4146268712cfc294dc0b00 in term 0.
I20260927 14:00:38.461233 8279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe32f8ba3f843cdb9bb56386a609b2b" candidate_uuid: "c02df5024e4146268712cfc294dc0b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cc5657efad4dc68a4387d6fae7634e" is_pre_election: true
I20260927 14:00:38.461715 8279 raft_consensus.cc:2454] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c02df5024e4146268712cfc294dc0b00 in term 0.
I20260927 14:00:38.461935 7951 leader_election.cc:302] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c02df5024e4146268712cfc294dc0b00, ddee7737520c416f8fef17e11a456768; no voters:
I20260927 14:00:38.462335 8350 raft_consensus.cc:2781] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:00:38.462509 8350 raft_consensus.cc:482] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:00:38.462625 8350 raft_consensus.cc:3037] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:38.465219 8350 raft_consensus.cc:504] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02df5024e4146268712cfc294dc0b00" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35311 } } peers { permanent_uuid: "ddee7737520c416f8fef17e11a456768" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 45121 } } peers { permanent_uuid: "96cc5657efad4dc68a4387d6fae7634e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 32787 } }
I20260927 14:00:38.465847 8350 leader_election.cc:288] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [CANDIDATE]: Term 1 election: Requested vote from peers ddee7737520c416f8fef17e11a456768 (127.22.26.194:45121), 96cc5657efad4dc68a4387d6fae7634e (127.22.26.195:32787)
I20260927 14:00:38.466336 8279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe32f8ba3f843cdb9bb56386a609b2b" candidate_uuid: "c02df5024e4146268712cfc294dc0b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cc5657efad4dc68a4387d6fae7634e"
I20260927 14:00:38.466369 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe32f8ba3f843cdb9bb56386a609b2b" candidate_uuid: "c02df5024e4146268712cfc294dc0b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddee7737520c416f8fef17e11a456768"
I20260927 14:00:38.466530 8279 raft_consensus.cc:3037] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:38.466595 8149 raft_consensus.cc:3037] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:38.468851 8279 raft_consensus.cc:2454] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c02df5024e4146268712cfc294dc0b00 in term 1.
I20260927 14:00:38.468946 8149 raft_consensus.cc:2454] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c02df5024e4146268712cfc294dc0b00 in term 1.
I20260927 14:00:38.469347 7952 leader_election.cc:302] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cc5657efad4dc68a4387d6fae7634e, c02df5024e4146268712cfc294dc0b00; no voters:
I20260927 14:00:38.469746 8350 raft_consensus.cc:2781] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:38.470214 8350 raft_consensus.cc:686] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [term 1 LEADER]: Becoming Leader. State: Replica: c02df5024e4146268712cfc294dc0b00, State: Running, Role: LEADER
I20260927 14:00:38.470685 8350 consensus_queue.cc:237] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02df5024e4146268712cfc294dc0b00" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35311 } } peers { permanent_uuid: "ddee7737520c416f8fef17e11a456768" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 45121 } } peers { permanent_uuid: "96cc5657efad4dc68a4387d6fae7634e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 32787 } }
I20260927 14:00:38.475066 7877 catalog_manager.cc:5495] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 reported cstate change: term changed from 0 to 1, leader changed from <none> to c02df5024e4146268712cfc294dc0b00 (127.22.26.193). New cstate: current_term: 1 leader_uuid: "c02df5024e4146268712cfc294dc0b00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02df5024e4146268712cfc294dc0b00" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 35311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddee7737520c416f8fef17e11a456768" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 45121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cc5657efad4dc68a4387d6fae7634e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 32787 } health_report { overall_health: UNKNOWN } } }
I20260927 14:00:38.495205 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.490094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.22.26.193:35311)
0927 14:00:38.490307 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:38.490312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:38.490331 (+ 19us) server_negotiation.cc:407] Connection header received
0927 14:00:38.490416 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:38.490419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:38.490513 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:38.490644 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:00:38.490962 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.491746 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.493066 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.493485 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.493586 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.494024 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:00:38.494564 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:00:38.494674 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:38.494774 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 14:00:38.494836 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 14:00:38.503650 8149 raft_consensus.cc:1261] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768 [term 1 FOLLOWER]: Refusing update from remote peer c02df5024e4146268712cfc294dc0b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:00:38.503669 8279 raft_consensus.cc:1261] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e [term 1 FOLLOWER]: Refusing update from remote peer c02df5024e4146268712cfc294dc0b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:00:38.504635 8353 consensus_queue.cc:1035] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddee7737520c416f8fef17e11a456768" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 45121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:00:38.504993 8350 consensus_queue.cc:1035] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cc5657efad4dc68a4387d6fae7634e" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 32787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:00:38.514510 8356 mvcc.cc:205] Tried to move back new op lower bound from 7333960247299973120 to 7333960247179993088. Current Snapshot: MvccSnapshot[applied={T|T < 7333960247299973120}]
I20260927 14:00:38.515571 8357 mvcc.cc:205] Tried to move back new op lower bound from 7333960247299973120 to 7333960247179993088. Current Snapshot: MvccSnapshot[applied={T|T < 7333960247299973120}]
I20260927 14:00:38.529763 7875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:00:38.530058 7875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:00:38.533870 7875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b638e7b695a74eb282b1bd707233dab1: Sending DeleteTablet for 3 replicas of tablet 8fe32f8ba3f843cdb9bb56386a609b2b
I20260927 14:00:38.534937 7998 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe32f8ba3f843cdb9bb56386a609b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:38 UTC) from {username='slave'} at 127.0.0.1:43044
I20260927 14:00:38.535033 8129 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe32f8ba3f843cdb9bb56386a609b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:38 UTC) from {username='slave'} at 127.0.0.1:56220
I20260927 14:00:38.535379 8259 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe32f8ba3f843cdb9bb56386a609b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:38 UTC) from {username='slave'} at 127.0.0.1:57034
I20260927 14:00:38.537164 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 7936
I20260927 14:00:38.543649 8366 tablet_replica.cc:330] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768: stopping tablet replica
I20260927 14:00:38.544145 8366 raft_consensus.cc:2229] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:00:38.544706 8366 raft_consensus.cc:2258] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:38.548522 8368 tablet_replica.cc:330] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e: stopping tablet replica
I20260927 14:00:38.549054 8368 raft_consensus.cc:2229] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:00:38.549587 8368 raft_consensus.cc:2258] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:38.550426 8367 tablet_replica.cc:330] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00: stopping tablet replica
I20260927 14:00:38.550843 8367 raft_consensus.cc:2229] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:00:38.551568 8367 raft_consensus.cc:2258] T 8fe32f8ba3f843cdb9bb56386a609b2b P c02df5024e4146268712cfc294dc0b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:38.552201 8368 ts_tablet_manager.cc:1905] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:38.553336 8366 ts_tablet_manager.cc:1905] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:38.554867 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.539341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.22.26.193:35311)
0927 14:00:38.539582 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:38.539591 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:38.544237 (+ 4646us) server_negotiation.cc:407] Connection header received
0927 14:00:38.544802 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:38.544812 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:38.544942 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:38.545059 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:38.545843 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.547503 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.550808 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.551530 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.551652 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.553154 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:38.553164 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:38.553184 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:38.553957 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:38.554016 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:38.554042 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:38.554118 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:38.554358 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 14:00:38.554484 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 14:00:38.556075 8058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:38.561429 8368 ts_tablet_manager.cc:1918] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:38.561674 8368 log.cc:1196] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/8fe32f8ba3f843cdb9bb56386a609b2b
I20260927 14:00:38.561837 8366 ts_tablet_manager.cc:1918] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:38.561987 8366 log.cc:1196] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/8fe32f8ba3f843cdb9bb56386a609b2b
I20260927 14:00:38.562289 8368 ts_tablet_manager.cc:1939] T 8fe32f8ba3f843cdb9bb56386a609b2b P 96cc5657efad4dc68a4387d6fae7634e: Deleting consensus metadata
I20260927 14:00:38.562494 8366 ts_tablet_manager.cc:1939] T 8fe32f8ba3f843cdb9bb56386a609b2b P ddee7737520c416f8fef17e11a456768: Deleting consensus metadata
I20260927 14:00:38.564335 7861 catalog_manager.cc:4841] TS ddee7737520c416f8fef17e11a456768 (127.22.26.194:45121): tablet 8fe32f8ba3f843cdb9bb56386a609b2b (table test-table [id=73f4d269bbce41579bd83c74f331ebc9]) successfully deleted
I20260927 14:00:38.564440 7862 catalog_manager.cc:4841] TS 96cc5657efad4dc68a4387d6fae7634e (127.22.26.195:32787): tablet 8fe32f8ba3f843cdb9bb56386a609b2b (table test-table [id=73f4d269bbce41579bd83c74f331ebc9]) successfully deleted
I20260927 14:00:38.653534 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 7936
W20260927 14:00:38.673311 7861 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.26.193:35311 (error 108)
I20260927 14:00:38.674043 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 8067
I20260927 14:00:38.675462 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.674876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35311 (local address 127.0.0.1:43084)
0927 14:00:38.675209 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:38.675326 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260927 14:00:38.675693 7861 catalog_manager.cc:4584] TS c02df5024e4146268712cfc294dc0b00 (127.22.26.193:35311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fe32f8ba3f843cdb9bb56386a609b2b: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35311: connect: Connection refused (error 111)
I20260927 14:00:38.682094 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.675030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.22.26.194:45121)
0927 14:00:38.675269 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:38.675275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:38.675297 (+ 22us) server_negotiation.cc:407] Connection header received
0927 14:00:38.675416 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:38.675423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:38.675564 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:38.675695 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:38.676205 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.677182 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.678644 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.679146 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.679284 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.680590 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:38.680601 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:38.680616 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:38.681242 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:38.681293 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:38.681315 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:38.681466 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:38.681642 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 14:00:38.681761 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 14:00:38.682941 8189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:38.729549 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.729065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35311 (local address 127.0.0.1:43100)
0927 14:00:38.729297 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:38.729386 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 14:00:38.730655 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.730336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35311 (local address 127.0.0.1:43106)
0927 14:00:38.730469 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:38.730539 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 14:00:38.772981 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 8067
I20260927 14:00:38.790377 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 8197
I20260927 14:00:38.791169 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.790601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35311 (local address 127.0.0.1:43122)
0927 14:00:38.790852 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:38.790937 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 14:00:38.792281 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.791934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35311 (local address 127.0.0.1:43138)
0927 14:00:38.792085 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:38.792150 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 14:00:38.797554 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.791301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.22.26.195:32787)
0927 14:00:38.791462 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:38.791468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:38.791490 (+ 22us) server_negotiation.cc:407] Connection header received
0927 14:00:38.791660 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:38.791665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:38.791784 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:38.791874 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:38.792347 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.793231 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.794627 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:38.795011 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:38.795216 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:38.796303 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:38.796312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:38.796327 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:38.796922 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:38.796969 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:38.796988 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:38.797084 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:38.797231 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 14:00:38.797326 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 14:00:38.798285 8319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:38.873112 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.872472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35311 (local address 127.0.0.1:43148)
0927 14:00:38.872744 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:38.872905 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260927 14:00:38.874306 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:38.873861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:35311 (local address 127.0.0.1:43156)
0927 14:00:38.874093 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:38.874182 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 14:00:38.888083 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 8197
I20260927 14:00:38.904870 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 7846
I20260927 14:00:38.905929 7907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:39.016987 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 7846
2026-09-27T14:00:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2678 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:00:39 dist-test-slave-gkks krb5kdc[8377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:00:39 dist-test-slave-gkks krb5kdc[8377](info): set up 2 sockets
Sep 27 14:00:39 dist-test-slave-gkks krb5kdc[8377](info): commencing operation
krb5kdc: starting...
W20260927 14:00:40.135425 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:00:40 dist-test-slave-gkks krb5kdc[8377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:00:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:00:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 14:00:40.387157 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:36157
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:36157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:40.716789 8390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:40.717144 8390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:40.717299 8390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:40.724474 8390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:00:40.724625 8390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:40.724736 8390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:40.724843 8390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:00:40.724916 8390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:00:40.734942 8390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:36157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:36157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:40.736402 8390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:40.737772 8390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:40.750671 8398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:40.750775 8396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:40.750716 8395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:40.756757 8390 server_base.cc:1034] running on GCE node
I20260927 14:00:40.757666 8390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:40.759332 8390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:40.760583 8390 hybrid_clock.cc:648] HybridClock initialized: now 1790517640760540 us; error 54 us; skew 500 ppm
I20260927 14:00:40.760953 8390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:40 dist-test-slave-gkks krb5kdc[8377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517640, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:40.764148 8390 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 14:00:40.765784 8390 webserver.cc:460] Webserver started at http://127.22.26.254:42793/ using document root <none> and password file <none>
I20260927 14:00:40.766512 8390 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:40.766659 8390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:40.766973 8390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:40.770119 8390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "bed8af9e966148a48e253d419741e7e2"
format_stamp: "Formatted at 2026-09-27 14:00:40 on dist-test-slave-gkks"
server_key: "e59c2a0848feddf7d802e5351aa74979"
server_key_iv: "51eb8f43db7a9908a0b4e4b03a351fff"
server_key_version: "encryptionkey@0"
I20260927 14:00:40.770889 8390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "bed8af9e966148a48e253d419741e7e2"
format_stamp: "Formatted at 2026-09-27 14:00:40 on dist-test-slave-gkks"
server_key: "e59c2a0848feddf7d802e5351aa74979"
server_key_iv: "51eb8f43db7a9908a0b4e4b03a351fff"
server_key_version: "encryptionkey@0"
I20260927 14:00:40.776262 8390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 14:00:40.780009 8405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:40.781447 8390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:00:40.781689 8390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "bed8af9e966148a48e253d419741e7e2"
format_stamp: "Formatted at 2026-09-27 14:00:40 on dist-test-slave-gkks"
server_key: "e59c2a0848feddf7d802e5351aa74979"
server_key_iv: "51eb8f43db7a9908a0b4e4b03a351fff"
server_key_version: "encryptionkey@0"
I20260927 14:00:40.781899 8390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:40.812079 8390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:40.816179 8390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:40.816484 8390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:40.835476 8390 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:36157
I20260927 14:00:40.835498 8456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:36157 every 8 connection(s)
I20260927 14:00:40.836879 8390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:00:40.840624 8457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:40.841413 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 8390
I20260927 14:00:40.841621 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:00:40.842108 22635 external_mini_cluster.cc:1428] Setting key cfb6002262d4f7ddf228cf1f308d6353
I20260927 14:00:40.855427 8457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2: Bootstrap starting.
Sep 27 14:00:40 dist-test-slave-gkks krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:00:40.858664 8457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:40.859997 8457 log.cc:824] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:40.862905 8457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2: No bootstrap required, opened a new log
I20260927 14:00:40.866560 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:40.844170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.22.26.254:36157)
0927 14:00:40.845041 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:40.845057 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:40.845114 (+ 57us) server_negotiation.cc:407] Connection header received
0927 14:00:40.847406 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:40.847437 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:40.847971 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:40.848440 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:40.849901 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:40.851958 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:40.853300 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:40.854340 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:40.854506 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:40.858986 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:40.858999 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:40.859014 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:40.859061 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:40.862287 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:40.863002 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:40.863012 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:40.863030 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:40.863259 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:40.863694 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:40.863699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:40.863703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:40.864243 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:40.864561 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:40.865065 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 14:00:40.865519 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":351,"threads_started":1}
I20260927 14:00:40.871305 8457 raft_consensus.cc:348] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed8af9e966148a48e253d419741e7e2" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36157 } }
I20260927 14:00:40.871718 8457 raft_consensus.cc:374] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:40.871842 8457 raft_consensus.cc:729] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed8af9e966148a48e253d419741e7e2, State: Initialized, Role: FOLLOWER
I20260927 14:00:40.872385 8457 consensus_queue.cc:260] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed8af9e966148a48e253d419741e7e2" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36157 } }
I20260927 14:00:40.872632 8457 raft_consensus.cc:388] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:00:40.872779 8457 raft_consensus.cc:482] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:00:40.872954 8457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:40.875200 8457 raft_consensus.cc:504] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed8af9e966148a48e253d419741e7e2" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36157 } }
I20260927 14:00:40.875694 8457 leader_election.cc:302] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bed8af9e966148a48e253d419741e7e2; no voters:
I20260927 14:00:40.876267 8457 leader_election.cc:288] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:00:40.876425 8462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:40.877228 8462 raft_consensus.cc:686] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [term 1 LEADER]: Becoming Leader. State: Replica: bed8af9e966148a48e253d419741e7e2, State: Running, Role: LEADER
I20260927 14:00:40.877746 8462 consensus_queue.cc:237] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed8af9e966148a48e253d419741e7e2" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36157 } }
I20260927 14:00:40.878449 8457 sys_catalog.cc:564] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:00:40.880692 8463 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bed8af9e966148a48e253d419741e7e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed8af9e966148a48e253d419741e7e2" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36157 } } }
I20260927 14:00:40.881186 8463 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:40.881723 8464 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bed8af9e966148a48e253d419741e7e2. Latest consensus state: current_term: 1 leader_uuid: "bed8af9e966148a48e253d419741e7e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed8af9e966148a48e253d419741e7e2" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36157 } } }
I20260927 14:00:40.882057 8464 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:40.888270 8470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:00:40.893923 8470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:00:40.903301 8470 catalog_manager.cc:1347] Generated new cluster ID: 5b88b60cbe7c41be884b27e80a15e719
I20260927 14:00:40.903426 8470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:00:40.931809 8470 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:00:40.933768 8470 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:00:40.940876 8470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2: Generated new TSK 0
I20260927 14:00:40.941759 8470 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 14:00:41.004920 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36157
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:41.335989 8483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:41.336253 8483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:41.336406 8483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:41.343436 8483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:41.343530 8483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:41.343736 8483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 14:00:41.355346 8483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:41.356668 8483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:41.358006 8483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:41.371680 8488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:41.371712 8491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:41.371632 8489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:41.372695 8483 server_base.cc:1034] running on GCE node
I20260927 14:00:41.373325 8483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:41.374277 8483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:41.375504 8483 hybrid_clock.cc:648] HybridClock initialized: now 1790517641375455 us; error 44 us; skew 500 ppm
I20260927 14:00:41.375833 8483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:41 dist-test-slave-gkks krb5kdc[8377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517641, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:41.379038 8483 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 14:00:41.380851 8483 webserver.cc:460] Webserver started at http://127.22.26.193:36527/ using document root <none> and password file <none>
I20260927 14:00:41.381640 8483 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:41.381768 8483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:41.382048 8483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:41.385337 8483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "555038538d634b9e83979fafa8c5b971"
format_stamp: "Formatted at 2026-09-27 14:00:41 on dist-test-slave-gkks"
server_key: "6a545cd7288978ad494975a483934c74"
server_key_iv: "71190bed97917a08513cf9fb2154d0c6"
server_key_version: "encryptionkey@0"
I20260927 14:00:41.386143 8483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "555038538d634b9e83979fafa8c5b971"
format_stamp: "Formatted at 2026-09-27 14:00:41 on dist-test-slave-gkks"
server_key: "6a545cd7288978ad494975a483934c74"
server_key_iv: "71190bed97917a08513cf9fb2154d0c6"
server_key_version: "encryptionkey@0"
I20260927 14:00:41.391644 8483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 14:00:41.395251 8498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:41.396562 8483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:00:41.396692 8483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "555038538d634b9e83979fafa8c5b971"
format_stamp: "Formatted at 2026-09-27 14:00:41 on dist-test-slave-gkks"
server_key: "6a545cd7288978ad494975a483934c74"
server_key_iv: "71190bed97917a08513cf9fb2154d0c6"
server_key_version: "encryptionkey@0"
I20260927 14:00:41.396862 8483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:41.415432 8483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:41.419363 8483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:41.419667 8483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:41.420959 8483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:41.422849 8483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:41.422950 8483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:41.423123 8483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:41.423207 8483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 14:00:41.459048 8483 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:39329
I20260927 14:00:41.459187 8610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:39329 every 8 connection(s)
I20260927 14:00:41.460422 8483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 14:00:41.462487 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 8483
I20260927 14:00:41.462697 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 14:00:41.463276 22635 external_mini_cluster.cc:1428] Setting key 407e76fd02a3528763635f8ea9b9665e
Sep 27 14:00:41 dist-test-slave-gkks krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517641, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:00:41.482306 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:41.463168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:36251 (local address 127.22.26.254:36157)
0927 14:00:41.463740 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:41.463746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:41.464782 (+ 1036us) server_negotiation.cc:407] Connection header received
0927 14:00:41.466841 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:41.466847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:41.466940 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:41.467059 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:41.468681 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:41.469406 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:41.470994 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:41.471462 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:41.471612 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:41.476090 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:41.476095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:41.476099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:41.476132 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:41.478965 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:41.479887 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:41.479892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:41.479901 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:41.480021 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:41.480458 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:41.480462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:41.480466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:41.480752 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:41.480912 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:41.481663 (+ 751us) server_negotiation.cc:299] Negotiation successful
0927 14:00:41.481940 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":282,"threads_started":1}
I20260927 14:00:41.483426 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:41.463779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:36157 (local address 127.22.26.193:36251)
0927 14:00:41.464595 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:41.464650 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:41.466550 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:41.467349 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:41.467358 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:41.467908 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:41.468506 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:41.468523 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:41.469575 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:41.469580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:41.470848 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:41.470861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:41.471701 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:41.471710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:41.471964 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:41.472755 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:00:41.472798 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:41.475762 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:41.479165 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:41.479174 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:41.479194 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:41.479754 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:41.480138 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:41.480143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:41.480147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:41.480335 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:41.480882 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:41.480890 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:41.481257 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 14:00:41.481534 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 14:00:41.482164 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":317,"threads_started":1}
I20260927 14:00:41.485188 8611 heartbeater.cc:347] Connected to a master server at 127.22.26.254:36157
I20260927 14:00:41.485569 8611 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:41.486565 8611 heartbeater.cc:511] Master 127.22.26.254:36157 requested a full tablet report, sending...
I20260927 14:00:41.489105 8422 ts_manager.cc:194] Registered new tserver with Master: 555038538d634b9e83979fafa8c5b971 (127.22.26.193:39329)
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
I20260927 14:00:41.490674 8422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:36251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 14:00:41.523453 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36157
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:41.856148 8617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:41.856442 8617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:41.856662 8617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:41.864281 8617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:41.864406 8617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:41.864626 8617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 14:00:41.874397 8617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:41.875810 8617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:41.877152 8617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:41.891366 8623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:41.891398 8622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:41.891330 8625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:41.892124 8617 server_base.cc:1034] running on GCE node
I20260927 14:00:41.892768 8617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:41.893723 8617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:41.895040 8617 hybrid_clock.cc:648] HybridClock initialized: now 1790517641894963 us; error 52 us; skew 500 ppm
I20260927 14:00:41.895444 8617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:41 dist-test-slave-gkks krb5kdc[8377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517641, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:41.904019 8617 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 14:00:41.905506 8617 webserver.cc:460] Webserver started at http://127.22.26.194:44773/ using document root <none> and password file <none>
I20260927 14:00:41.906257 8617 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:41.906359 8617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:41.906620 8617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:41.909545 8617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "d9865ea1ebfa429faa6f3dafca5268eb"
format_stamp: "Formatted at 2026-09-27 14:00:41 on dist-test-slave-gkks"
server_key: "789c7ad5e584dcd37c87c082d9a34ac9"
server_key_iv: "f7652acaeae0fd64f742c7481e4d01a7"
server_key_version: "encryptionkey@0"
I20260927 14:00:41.910247 8617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "d9865ea1ebfa429faa6f3dafca5268eb"
format_stamp: "Formatted at 2026-09-27 14:00:41 on dist-test-slave-gkks"
server_key: "789c7ad5e584dcd37c87c082d9a34ac9"
server_key_iv: "f7652acaeae0fd64f742c7481e4d01a7"
server_key_version: "encryptionkey@0"
I20260927 14:00:41.915707 8617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 14:00:41.919179 8632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:41.920585 8617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:00:41.920737 8617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "d9865ea1ebfa429faa6f3dafca5268eb"
format_stamp: "Formatted at 2026-09-27 14:00:41 on dist-test-slave-gkks"
server_key: "789c7ad5e584dcd37c87c082d9a34ac9"
server_key_iv: "f7652acaeae0fd64f742c7481e4d01a7"
server_key_version: "encryptionkey@0"
I20260927 14:00:41.920895 8617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:41.941111 8617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:41.945415 8617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:41.945642 8617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:41.946820 8617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:41.948112 8617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:41.948187 8617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:41.948284 8617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:41.948333 8617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:41.981894 8617 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:35277
I20260927 14:00:41.981977 8744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:35277 every 8 connection(s)
I20260927 14:00:41.983444 8617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 14:00:41.987439 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 8617
I20260927 14:00:41.987640 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 14:00:41.988104 22635 external_mini_cluster.cc:1428] Setting key 52b650ffcfaef6f956adeaa8f38960e3
Sep 27 14:00:41 dist-test-slave-gkks krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517641, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:00:42.008242 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:41.986448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:33333 (local address 127.22.26.254:36157)
0927 14:00:41.987039 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:41.987046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:41.988540 (+ 1494us) server_negotiation.cc:407] Connection header received
0927 14:00:41.990750 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:41.990757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:41.990841 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:41.990963 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:41.992610 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:41.993862 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:41.995573 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:41.996358 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:41.996500 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.000656 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:42.000674 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:42.000682 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:42.000730 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:42.004507 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:42.005264 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:42.005271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:42.005277 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:42.005439 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:42.005856 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:42.005863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:42.005869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:42.006259 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:42.006412 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:42.007414 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0927 14:00:42.007710 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":311,"threads_started":1}
I20260927 14:00:42.009071 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:41.987413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:36157 (local address 127.22.26.194:33333)
0927 14:00:41.988341 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:41.988388 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:41.990478 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:41.991266 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:41.991276 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:41.991848 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:41.992431 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:41.992449 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:41.994056 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:41.994061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:41.995430 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:41.995442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:41.996569 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:41.996574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:41.996775 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:41.997483 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:00:41.997511 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:42.000409 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:42.004655 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:42.004665 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:42.004680 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:42.005112 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:42.005547 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:42.005550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:42.005552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:42.005737 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:42.006382 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:42.006388 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:42.006810 (+ 422us) client_negotiation.cc:769] Sending connection context
0927 14:00:42.007304 (+ 494us) client_negotiation.cc:240] Negotiation successful
0927 14:00:42.007782 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":427,"threads_started":1}
I20260927 14:00:42.010877 8745 heartbeater.cc:347] Connected to a master server at 127.22.26.254:36157
I20260927 14:00:42.011294 8745 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:42.012090 8745 heartbeater.cc:511] Master 127.22.26.254:36157 requested a full tablet report, sending...
I20260927 14:00:42.013787 8422 ts_manager.cc:194] Registered new tserver with Master: d9865ea1ebfa429faa6f3dafca5268eb (127.22.26.194:35277)
I20260927 14:00:42.014662 8422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:33333
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 14:00:42.048943 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36157
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:42.376610 8751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:42.376873 8751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:42.377027 8751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:42.384661 8751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:42.384773 8751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:42.384943 8751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 14:00:42.395011 8751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:42.396294 8751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:42.397603 8751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:42.410555 8756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:42.410604 8759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:42.410640 8757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:42.411922 8751 server_base.cc:1034] running on GCE node
I20260927 14:00:42.412482 8751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:42.413430 8751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:42.414654 8751 hybrid_clock.cc:648] HybridClock initialized: now 1790517642414637 us; error 68 us; skew 500 ppm
I20260927 14:00:42.414961 8751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:42 dist-test-slave-gkks krb5kdc[8377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517642, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:42.418063 8751 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 14:00:42.419826 8751 webserver.cc:460] Webserver started at http://127.22.26.195:42025/ using document root <none> and password file <none>
I20260927 14:00:42.420470 8751 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:42.420610 8751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:42.420920 8751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:42.423732 8751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77"
format_stamp: "Formatted at 2026-09-27 14:00:42 on dist-test-slave-gkks"
server_key: "35663f4574f99adee6ca755becbb9dfa"
server_key_iv: "f21da16cd5f77d6900bf203cfb529559"
server_key_version: "encryptionkey@0"
I20260927 14:00:42.424506 8751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77"
format_stamp: "Formatted at 2026-09-27 14:00:42 on dist-test-slave-gkks"
server_key: "35663f4574f99adee6ca755becbb9dfa"
server_key_iv: "f21da16cd5f77d6900bf203cfb529559"
server_key_version: "encryptionkey@0"
I20260927 14:00:42.429396 8751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:00:42.432616 8766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:42.433827 8751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:00:42.433990 8751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77"
format_stamp: "Formatted at 2026-09-27 14:00:42 on dist-test-slave-gkks"
server_key: "35663f4574f99adee6ca755becbb9dfa"
server_key_iv: "f21da16cd5f77d6900bf203cfb529559"
server_key_version: "encryptionkey@0"
I20260927 14:00:42.434171 8751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:42.451213 8751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:42.454835 8751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:42.455153 8751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:42.456061 8751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:42.457302 8751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:42.457403 8751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:42.457546 8751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:42.457618 8751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:42.491348 8751 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:43167
I20260927 14:00:42.491412 8878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:43167 every 8 connection(s)
I20260927 14:00:42.492712 8751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 14:00:42.494462 8611 heartbeater.cc:503] Master 127.22.26.254:36157 was elected leader, sending a full tablet report...
I20260927 14:00:42.494618 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 8751
I20260927 14:00:42.494825 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 14:00:42.495400 22635 external_mini_cluster.cc:1428] Setting key 1f4c156f5ed3b0f4cce05f71c691b7d0
Sep 27 14:00:42 dist-test-slave-gkks krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517642, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:00:42.514112 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:42.495535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:49171 (local address 127.22.26.254:36157)
0927 14:00:42.495738 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:42.495744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:42.497440 (+ 1696us) server_negotiation.cc:407] Connection header received
0927 14:00:42.499527 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:42.499531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:42.499614 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:42.499745 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:42.501549 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.502297 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.503839 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.504238 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.504330 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.507874 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:42.507879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:42.507882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:42.507911 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:42.511200 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:42.511882 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:42.511887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:42.511891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:42.512021 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:42.512441 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:42.512446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:42.512449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:42.512732 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:42.512894 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:42.513556 (+ 662us) server_negotiation.cc:299] Negotiation successful
0927 14:00:42.513788 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 14:00:42.515143 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:42.496244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:36157 (local address 127.22.26.195:49171)
0927 14:00:42.497283 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:42.497336 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:42.499296 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:42.499971 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:42.499980 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:42.500493 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:42.501348 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:42.501381 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.502436 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.502440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:42.503721 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:42.503733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.504410 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.504414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:42.504606 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.505220 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:00:42.505246 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:42.507678 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:42.511331 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:42.511341 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:42.511362 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:42.511753 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:42.512132 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:42.512138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:42.512141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:42.512323 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:42.512858 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:42.512866 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:42.513199 (+ 333us) client_negotiation.cc:769] Sending connection context
0927 14:00:42.513457 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 14:00:42.513958 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":440,"threads_started":1}
I20260927 14:00:42.516479 8879 heartbeater.cc:347] Connected to a master server at 127.22.26.254:36157
I20260927 14:00:42.516775 8879 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:42.517436 8879 heartbeater.cc:511] Master 127.22.26.254:36157 requested a full tablet report, sending...
I20260927 14:00:42.518744 8422 ts_manager.cc:194] Registered new tserver with Master: a8fb72a1b4ff44bea5dbe3da15c6ad77 (127.22.26.195:43167)
I20260927 14:00:42.519613 8422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:49171
I20260927 14:00:42.522815 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:00:42.540495 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:42.527812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58434 (local address 127.22.26.254:36157)
0927 14:00:42.528046 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:42.528053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:42.528363 (+ 310us) server_negotiation.cc:407] Connection header received
0927 14:00:42.528500 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:42.528503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:42.528570 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:42.528649 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:42.530009 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.530740 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.532096 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.532497 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.532588 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.535055 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:42.535059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:42.535062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:42.535141 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:42.537870 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:42.538678 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:42.538682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:42.538684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:42.538850 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:42.539278 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:42.539281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:42.539283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:42.539554 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:42.539778 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:42.540016 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 14:00:42.540138 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 14:00:42.544464 8422 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:00:42.551764 8422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:00:42.583907 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:42.563732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.22.26.193:39329)
0927 14:00:42.575569 (+ 11837us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:42.575580 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:42.575612 (+ 32us) server_negotiation.cc:407] Connection header received
0927 14:00:42.575747 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:42.575759 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:42.576036 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:42.576989 (+ 953us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:42.577273 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.578661 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.581043 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.582665 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.582835 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.583117 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:42.583410 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 14:00:42.583586 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11660,"thread_start_us":12435,"threads_started":1}
I20260927 14:00:42.583906 8891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:42.563233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:39329 (local address 127.0.0.1:57894)
0927 14:00:42.575403 (+ 12170us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:42.575420 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:42.575526 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:42.576473 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:42.576481 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:42.576518 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:42.577038 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:42.577056 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.578783 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.578791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:42.580874 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:42.580887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.582896 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.582902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:42.583160 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.583210 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 14:00:42.583366 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 14:00:42.583558 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12043,"thread_start_us":11107,"threads_started":1}
I20260927 14:00:42.586232 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:42.576000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.22.26.195:43167)
0927 14:00:42.577731 (+ 1731us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:42.577739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:42.578444 (+ 705us) server_negotiation.cc:407] Connection header received
0927 14:00:42.578624 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:42.578632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:42.578842 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:42.579030 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:42.579701 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.580983 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.583527 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.585040 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.585183 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.585464 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:42.585679 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 14:00:42.585873 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1579,"thread_start_us":1451,"threads_started":1}
I20260927 14:00:42.586382 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:42.577773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:43167 (local address 127.0.0.1:38012)
0927 14:00:42.578326 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:42.578344 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:42.578458 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:42.579036 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:42.579040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:42.579064 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:42.579491 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:42.579502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.581112 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.581119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:42.583351 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:42.583365 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.585287 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.585294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:42.585475 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.585510 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 14:00:42.585625 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 14:00:42.585770 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":43,"thread_start_us":253,"threads_started":1}
I20260927 14:00:42.586975 8892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:42.563131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:35277 (local address 127.0.0.1:45152)
0927 14:00:42.574727 (+ 11596us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:42.574764 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:42.574931 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:42.576844 (+ 1913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:42.576854 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:42.576889 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:42.578656 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:42.578670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.580524 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.580529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:42.582810 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:42.582830 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.585617 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.585625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:42.585829 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.585869 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 14:00:42.585978 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 14:00:42.586207 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11343,"mutex_wait_us":1,"thread_start_us":11970,"threads_started":1}
I20260927 14:00:42.587813 8893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:42.563311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.22.26.194:35277)
0927 14:00:42.576010 (+ 12699us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:42.576019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:42.576045 (+ 26us) server_negotiation.cc:407] Connection header received
0927 14:00:42.576141 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:42.576151 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:42.576402 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:42.576586 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:42.578845 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.580385 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.583002 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.585082 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.585212 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.585392 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:42.587277 (+ 1885us) server_negotiation.cc:299] Negotiation successful
0927 14:00:42.587451 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12537,"thread_start_us":12437,"threads_started":1}
I20260927 14:00:42.588303 8546 tablet_service.cc:1467] Processing CreateTablet for tablet 8885e3dade3d4937ba8e34158c87a725 (DEFAULT_TABLE table=test-table [id=b0c6abd31d62478dbc10656b3ef5f038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:42.590495 8546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8885e3dade3d4937ba8e34158c87a725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:42.590123 8814 tablet_service.cc:1467] Processing CreateTablet for tablet 8885e3dade3d4937ba8e34158c87a725 (DEFAULT_TABLE table=test-table [id=b0c6abd31d62478dbc10656b3ef5f038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:42.591470 8814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8885e3dade3d4937ba8e34158c87a725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:42.591459 8680 tablet_service.cc:1467] Processing CreateTablet for tablet 8885e3dade3d4937ba8e34158c87a725 (DEFAULT_TABLE table=test-table [id=b0c6abd31d62478dbc10656b3ef5f038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:42.592787 8680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8885e3dade3d4937ba8e34158c87a725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:42.602283 8897 tablet_bootstrap.cc:492] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77: Bootstrap starting.
I20260927 14:00:42.606087 8897 tablet_bootstrap.cc:654] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:42.607365 8897 log.cc:824] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:42.607813 8898 tablet_bootstrap.cc:492] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971: Bootstrap starting.
I20260927 14:00:42.611835 8899 tablet_bootstrap.cc:492] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb: Bootstrap starting.
I20260927 14:00:42.612262 8898 tablet_bootstrap.cc:654] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:42.613793 8898 log.cc:824] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:42.614918 8899 tablet_bootstrap.cc:654] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:42.616228 8899 log.cc:824] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:42.616616 8897 tablet_bootstrap.cc:492] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77: No bootstrap required, opened a new log
I20260927 14:00:42.616925 8897 ts_tablet_manager.cc:1397] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260927 14:00:42.617187 8898 tablet_bootstrap.cc:492] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971: No bootstrap required, opened a new log
I20260927 14:00:42.617581 8898 ts_tablet_manager.cc:1397] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 14:00:42.620493 8899 tablet_bootstrap.cc:492] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb: No bootstrap required, opened a new log
I20260927 14:00:42.620750 8899 ts_tablet_manager.cc:1397] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 14:00:42.624779 8897 raft_consensus.cc:348] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9865ea1ebfa429faa6f3dafca5268eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35277 } } peers { permanent_uuid: "555038538d634b9e83979fafa8c5b971" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39329 } } peers { permanent_uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43167 } }
I20260927 14:00:42.625197 8897 raft_consensus.cc:374] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:42.625296 8897 raft_consensus.cc:729] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fb72a1b4ff44bea5dbe3da15c6ad77, State: Initialized, Role: FOLLOWER
I20260927 14:00:42.625909 8897 consensus_queue.cc:260] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9865ea1ebfa429faa6f3dafca5268eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35277 } } peers { permanent_uuid: "555038538d634b9e83979fafa8c5b971" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39329 } } peers { permanent_uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43167 } }
I20260927 14:00:42.626968 8879 heartbeater.cc:503] Master 127.22.26.254:36157 was elected leader, sending a full tablet report...
I20260927 14:00:42.627704 8897 ts_tablet_manager.cc:1428] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260927 14:00:42.628553 8898 raft_consensus.cc:348] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9865ea1ebfa429faa6f3dafca5268eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35277 } } peers { permanent_uuid: "555038538d634b9e83979fafa8c5b971" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39329 } } peers { permanent_uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43167 } }
I20260927 14:00:42.628990 8898 raft_consensus.cc:374] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:42.629102 8898 raft_consensus.cc:729] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555038538d634b9e83979fafa8c5b971, State: Initialized, Role: FOLLOWER
I20260927 14:00:42.629714 8898 consensus_queue.cc:260] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9865ea1ebfa429faa6f3dafca5268eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35277 } } peers { permanent_uuid: "555038538d634b9e83979fafa8c5b971" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39329 } } peers { permanent_uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43167 } }
I20260927 14:00:42.631513 8899 raft_consensus.cc:348] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9865ea1ebfa429faa6f3dafca5268eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35277 } } peers { permanent_uuid: "555038538d634b9e83979fafa8c5b971" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39329 } } peers { permanent_uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43167 } }
I20260927 14:00:42.631995 8899 raft_consensus.cc:374] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:42.632128 8899 raft_consensus.cc:729] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9865ea1ebfa429faa6f3dafca5268eb, State: Initialized, Role: FOLLOWER
I20260927 14:00:42.632325 8898 ts_tablet_manager.cc:1428] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 14:00:42.632692 8899 consensus_queue.cc:260] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9865ea1ebfa429faa6f3dafca5268eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35277 } } peers { permanent_uuid: "555038538d634b9e83979fafa8c5b971" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39329 } } peers { permanent_uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43167 } }
I20260927 14:00:42.633697 8745 heartbeater.cc:503] Master 127.22.26.254:36157 was elected leader, sending a full tablet report...
I20260927 14:00:42.634322 8899 ts_tablet_manager.cc:1428] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260927 14:00:42.712774 8612 tablet.cc:2369] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:00:42.735528 8746 tablet.cc:2369] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:00:42.744371 8880 tablet.cc:2369] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:00:42.955953 8904 raft_consensus.cc:482] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:00:42.956298 8904 raft_consensus.cc:504] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9865ea1ebfa429faa6f3dafca5268eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35277 } } peers { permanent_uuid: "555038538d634b9e83979fafa8c5b971" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39329 } } peers { permanent_uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43167 } }
I20260927 14:00:42.958796 8904 leader_election.cc:288] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9865ea1ebfa429faa6f3dafca5268eb (127.22.26.194:35277), a8fb72a1b4ff44bea5dbe3da15c6ad77 (127.22.26.195:43167)
I20260927 14:00:42.968183 8893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:42.959279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:44103 (local address 127.22.26.194:35277)
0927 14:00:42.959514 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:42.959521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:42.960485 (+ 964us) server_negotiation.cc:407] Connection header received
0927 14:00:42.960871 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:42.960879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:42.960985 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:42.961099 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:42.961940 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.962969 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.965761 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.966986 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.967140 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.967223 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:42.967779 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 14:00:42.967915 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 14:00:42.968966 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:42.959425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:35277 (local address 127.22.26.193:44103)
0927 14:00:42.960157 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:42.960178 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:42.960321 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:42.961138 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:42.961145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:42.961200 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:42.961729 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:42.961746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.963116 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.963122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:42.965558 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:42.965576 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.967396 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.967403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:42.967579 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.967615 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 14:00:42.968386 (+ 771us) client_negotiation.cc:240] Negotiation successful
0927 14:00:42.968530 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":412,"threads_started":1}
I20260927 14:00:42.969703 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:42.960623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:54113 (local address 127.22.26.195:43167)
0927 14:00:42.960823 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:42.960831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:42.962201 (+ 1370us) server_negotiation.cc:407] Connection header received
0927 14:00:42.962476 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:42.962482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:42.962557 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:42.962648 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:42.963647 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.964814 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.967204 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.968409 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.968536 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.968600 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:42.969328 (+ 728us) server_negotiation.cc:299] Negotiation successful
0927 14:00:42.969453 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 14:00:42.969704 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:42.960313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:43167 (local address 127.22.26.193:54113)
0927 14:00:42.961820 (+ 1507us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:42.961840 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:42.962319 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:42.962735 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:42.962742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:42.962769 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:42.963225 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:42.963244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.964989 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.964994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:42.966999 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:42.967013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:42.968940 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:42.968949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:42.969144 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:42.969174 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 14:00:42.969292 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 14:00:42.969407 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1363,"thread_start_us":1119,"threads_started":1}
I20260927 14:00:42.970155 8700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8885e3dade3d4937ba8e34158c87a725" candidate_uuid: "555038538d634b9e83979fafa8c5b971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9865ea1ebfa429faa6f3dafca5268eb" is_pre_election: true
I20260927 14:00:42.970736 8834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8885e3dade3d4937ba8e34158c87a725" candidate_uuid: "555038538d634b9e83979fafa8c5b971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77" is_pre_election: true
I20260927 14:00:42.971029 8700 raft_consensus.cc:2454] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 555038538d634b9e83979fafa8c5b971 in term 0.
I20260927 14:00:42.971316 8834 raft_consensus.cc:2454] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 555038538d634b9e83979fafa8c5b971 in term 0.
I20260927 14:00:42.971977 8500 leader_election.cc:302] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555038538d634b9e83979fafa8c5b971, a8fb72a1b4ff44bea5dbe3da15c6ad77; no voters:
I20260927 14:00:42.972532 8904 raft_consensus.cc:2781] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:00:42.972718 8904 raft_consensus.cc:482] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:00:42.972834 8904 raft_consensus.cc:3037] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:42.975332 8904 raft_consensus.cc:504] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9865ea1ebfa429faa6f3dafca5268eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35277 } } peers { permanent_uuid: "555038538d634b9e83979fafa8c5b971" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39329 } } peers { permanent_uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43167 } }
I20260927 14:00:42.976089 8904 leader_election.cc:288] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [CANDIDATE]: Term 1 election: Requested vote from peers d9865ea1ebfa429faa6f3dafca5268eb (127.22.26.194:35277), a8fb72a1b4ff44bea5dbe3da15c6ad77 (127.22.26.195:43167)
I20260927 14:00:42.976485 8700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8885e3dade3d4937ba8e34158c87a725" candidate_uuid: "555038538d634b9e83979fafa8c5b971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9865ea1ebfa429faa6f3dafca5268eb"
I20260927 14:00:42.976701 8700 raft_consensus.cc:3037] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:42.977106 8834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8885e3dade3d4937ba8e34158c87a725" candidate_uuid: "555038538d634b9e83979fafa8c5b971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77"
I20260927 14:00:42.977329 8834 raft_consensus.cc:3037] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:42.980199 8700 raft_consensus.cc:2454] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 555038538d634b9e83979fafa8c5b971 in term 1.
I20260927 14:00:42.980621 8500 leader_election.cc:302] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555038538d634b9e83979fafa8c5b971, d9865ea1ebfa429faa6f3dafca5268eb; no voters:
I20260927 14:00:42.980836 8834 raft_consensus.cc:2454] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 555038538d634b9e83979fafa8c5b971 in term 1.
I20260927 14:00:42.984959 8904 raft_consensus.cc:2781] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:42.991189 8904 raft_consensus.cc:686] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [term 1 LEADER]: Becoming Leader. State: Replica: 555038538d634b9e83979fafa8c5b971, State: Running, Role: LEADER
I20260927 14:00:42.991829 8904 consensus_queue.cc:237] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9865ea1ebfa429faa6f3dafca5268eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35277 } } peers { permanent_uuid: "555038538d634b9e83979fafa8c5b971" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39329 } } peers { permanent_uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43167 } }
I20260927 14:00:43.001076 8421 catalog_manager.cc:5495] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 555038538d634b9e83979fafa8c5b971 (127.22.26.193). New cstate: current_term: 1 leader_uuid: "555038538d634b9e83979fafa8c5b971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9865ea1ebfa429faa6f3dafca5268eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "555038538d634b9e83979fafa8c5b971" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 39329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43167 } health_report { overall_health: UNKNOWN } } }
I20260927 14:00:43.033020 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:43.027433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.22.26.193:39329)
0927 14:00:43.027652 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:43.027658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:43.027676 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:00:43.027879 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:43.027883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:43.027974 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:43.028128 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:00:43.028562 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:43.029301 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:43.030700 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:43.031135 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:43.031273 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:43.031893 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:00:43.032395 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:00:43.032578 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:43.032740 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 14:00:43.032790 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 14:00:43.041579 8700 raft_consensus.cc:1261] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb [term 1 FOLLOWER]: Refusing update from remote peer 555038538d634b9e83979fafa8c5b971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:00:43.041611 8834 raft_consensus.cc:1261] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77 [term 1 FOLLOWER]: Refusing update from remote peer 555038538d634b9e83979fafa8c5b971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:00:43.042691 8908 consensus_queue.cc:1035] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9865ea1ebfa429faa6f3dafca5268eb" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:00:43.043068 8904 consensus_queue.cc:1035] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8fb72a1b4ff44bea5dbe3da15c6ad77" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 43167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:00:43.054462 8911 mvcc.cc:205] Tried to move back new op lower bound from 7333960265887588352 to 7333960265699094528. Current Snapshot: MvccSnapshot[applied={T|T < 7333960265887588352}]
I20260927 14:00:43.058665 8912 mvcc.cc:205] Tried to move back new op lower bound from 7333960265887588352 to 7333960265699094528. Current Snapshot: MvccSnapshot[applied={T|T < 7333960265887588352}]
I20260927 14:00:43.068081 8422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:00:43.068374 8422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:00:43.071769 8422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bed8af9e966148a48e253d419741e7e2: Sending DeleteTablet for 3 replicas of tablet 8885e3dade3d4937ba8e34158c87a725
I20260927 14:00:43.072870 8680 tablet_service.cc:1514] Processing DeleteTablet for tablet 8885e3dade3d4937ba8e34158c87a725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:43 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:45152
I20260927 14:00:43.072919 8546 tablet_service.cc:1514] Processing DeleteTablet for tablet 8885e3dade3d4937ba8e34158c87a725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:43 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:57894
I20260927 14:00:43.073963 8814 tablet_service.cc:1514] Processing DeleteTablet for tablet 8885e3dade3d4937ba8e34158c87a725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:43 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:38012
I20260927 14:00:43.077497 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 8483
I20260927 14:00:43.085466 8921 tablet_replica.cc:330] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971: stopping tablet replica
I20260927 14:00:43.086103 8921 raft_consensus.cc:2229] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:00:43.086860 8921 raft_consensus.cc:2258] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:43.089493 8921 ts_tablet_manager.cc:1905] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:43.090116 8922 tablet_replica.cc:330] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb: stopping tablet replica
I20260927 14:00:43.090693 8922 raft_consensus.cc:2229] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:00:43.099309 8921 ts_tablet_manager.cc:1918] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:43.099902 8922 raft_consensus.cc:2258] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:43.102236 8922 ts_tablet_manager.cc:1905] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:43.103209 8923 tablet_replica.cc:330] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77: stopping tablet replica
I20260927 14:00:43.103240 8921 log.cc:1196] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/8885e3dade3d4937ba8e34158c87a725
I20260927 14:00:43.103818 8921 ts_tablet_manager.cc:1939] T 8885e3dade3d4937ba8e34158c87a725 P 555038538d634b9e83979fafa8c5b971: Deleting consensus metadata
I20260927 14:00:43.105698 8409 catalog_manager.cc:4841] TS 555038538d634b9e83979fafa8c5b971 (127.22.26.193:39329): tablet 8885e3dade3d4937ba8e34158c87a725 (table test-table [id=b0c6abd31d62478dbc10656b3ef5f038]) successfully deleted
I20260927 14:00:43.106508 8923 raft_consensus.cc:2229] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:00:43.107025 8923 raft_consensus.cc:2258] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 14:00:43 dist-test-slave-gkks krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 14:00:43.109027 8923 ts_tablet_manager.cc:1905] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:43.109843 8922 ts_tablet_manager.cc:1918] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:43.110036 8922 log.cc:1196] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/8885e3dade3d4937ba8e34158c87a725
I20260927 14:00:43.110579 8922 ts_tablet_manager.cc:1939] T 8885e3dade3d4937ba8e34158c87a725 P d9865ea1ebfa429faa6f3dafca5268eb: Deleting consensus metadata
I20260927 14:00:43.112488 8407 catalog_manager.cc:4841] TS d9865ea1ebfa429faa6f3dafca5268eb (127.22.26.194:35277): tablet 8885e3dade3d4937ba8e34158c87a725 (table test-table [id=b0c6abd31d62478dbc10656b3ef5f038]) successfully deleted
I20260927 14:00:43.116535 8923 ts_tablet_manager.cc:1918] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:43.116712 8923 log.cc:1196] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/8885e3dade3d4937ba8e34158c87a725
I20260927 14:00:43.116905 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:43.079026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.22.26.193:39329)
0927 14:00:43.079267 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:43.079276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:43.095034 (+ 15758us) server_negotiation.cc:407] Connection header received
0927 14:00:43.095558 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:43.095563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:43.095647 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:43.095725 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:43.097580 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:43.098859 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:43.100997 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:43.101702 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:43.101804 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:43.109908 (+ 8104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:43.109924 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:43.109945 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:43.110003 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:43.113721 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:43.114452 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:43.114458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:43.114461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:43.114612 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:43.115019 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:43.115023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:43.115025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:43.115437 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:43.115742 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:43.116063 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 14:00:43.116307 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 14:00:43.117233 8923 ts_tablet_manager.cc:1939] T 8885e3dade3d4937ba8e34158c87a725 P a8fb72a1b4ff44bea5dbe3da15c6ad77: Deleting consensus metadata
I20260927 14:00:43.117641 8606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:43.119012 8407 catalog_manager.cc:4841] TS a8fb72a1b4ff44bea5dbe3da15c6ad77 (127.22.26.195:43167): tablet 8885e3dade3d4937ba8e34158c87a725 (table test-table [id=b0c6abd31d62478dbc10656b3ef5f038]) successfully deleted
I20260927 14:00:43.217005 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 8483
I20260927 14:00:43.235917 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 8617
Sep 27 14:00:43 dist-test-slave-gkks krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 14:00:43.250870 8893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:43.236729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.22.26.194:35277)
0927 14:00:43.236912 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:43.236917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:43.236935 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:00:43.237079 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:43.237083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:43.237150 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:43.237248 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:43.238610 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:43.239685 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:43.240948 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:43.241373 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:43.241467 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:43.245670 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:43.245686 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:43.245701 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:43.245735 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:43.248624 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:43.249217 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:43.249224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:43.249226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:43.249357 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:43.249658 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:43.249661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:43.249663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:43.250009 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:43.250240 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:43.250449 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 14:00:43.250610 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 14:00:43.251554 8740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:43.342046 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 8617
I20260927 14:00:43.359732 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 8751
Sep 27 14:00:43 dist-test-slave-gkks krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 14:00:43.375511 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:43.360650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.22.26.195:43167)
0927 14:00:43.360832 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:43.360839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:43.360891 (+ 52us) server_negotiation.cc:407] Connection header received
0927 14:00:43.361167 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:43.361173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:43.361279 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:43.361380 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:43.362865 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:43.363719 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:43.365426 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:43.365916 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:43.366014 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:43.370217 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:43.370233 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:43.370250 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:43.370281 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:43.373175 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:43.373782 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:43.373789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:43.373791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:43.373924 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:43.374228 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:43.374231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:43.374233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:43.374567 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:43.374797 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:43.375010 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 14:00:43.375259 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 14:00:43.376157 8874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:43.469234 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 8751
I20260927 14:00:43.486080 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 8390
I20260927 14:00:43.487306 8452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:43.599500 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 8390
2026-09-27T14:00:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4582 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T14:00:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:00:43Z Disabled control of system clock
I20260927 14:00:43.675976 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:33843
--webserver_interface=127.22.26.254
--webserver_port=0
--builtin_ntp_servers=127.22.26.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:33843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 14:00:44.007557 8932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:44.007839 8932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:44.007989 8932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:44.015113 8932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:00:44.015215 8932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:44.015311 8932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:44.015429 8932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:00:44.015480 8932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:00:44.024690 8932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:33843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:33843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:44.026048 8932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:44.027940 8932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:44.039723 8937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:44.039669 8940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:44.039746 8938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:44.041010 8932 server_base.cc:1034] running on GCE node
I20260927 14:00:44.041795 8932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:44.043385 8932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:44.044615 8932 hybrid_clock.cc:648] HybridClock initialized: now 1790517644044574 us; error 59 us; skew 500 ppm
I20260927 14:00:44.044962 8932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 14:00:44.047269 8932 webserver.cc:460] Webserver started at http://127.22.26.254:45677/ using document root <none> and password file <none>
I20260927 14:00:44.047953 8932 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:44.048090 8932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:44.048417 8932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:44.051306 8932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "c0ced684ddc24db8bf5b779f95a1a78c"
format_stamp: "Formatted at 2026-09-27 14:00:44 on dist-test-slave-gkks"
server_key: "476ee0226af4973da857f710ac5fcd8f"
server_key_iv: "31c7c224b8f08073d6ce00336e8bfe89"
server_key_version: "encryptionkey@0"
I20260927 14:00:44.052109 8932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "c0ced684ddc24db8bf5b779f95a1a78c"
format_stamp: "Formatted at 2026-09-27 14:00:44 on dist-test-slave-gkks"
server_key: "476ee0226af4973da857f710ac5fcd8f"
server_key_iv: "31c7c224b8f08073d6ce00336e8bfe89"
server_key_version: "encryptionkey@0"
I20260927 14:00:44.057081 8932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 14:00:44.060547 8946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:44.061811 8932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:00:44.061998 8932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "c0ced684ddc24db8bf5b779f95a1a78c"
format_stamp: "Formatted at 2026-09-27 14:00:44 on dist-test-slave-gkks"
server_key: "476ee0226af4973da857f710ac5fcd8f"
server_key_iv: "31c7c224b8f08073d6ce00336e8bfe89"
server_key_version: "encryptionkey@0"
I20260927 14:00:44.062215 8932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:44.074707 8932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:44.075546 8932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:44.075824 8932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:44.094503 8932 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:33843
I20260927 14:00:44.094523 8997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:33843 every 8 connection(s)
I20260927 14:00:44.096060 8932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:00:44.098034 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 8932
I20260927 14:00:44.098240 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:00:44.098694 22635 external_mini_cluster.cc:1428] Setting key 6d44ca0840debd17827ddd3a8675e7a5
I20260927 14:00:44.100080 8998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:44.112061 9001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:44.100557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.22.26.254:33843)
0927 14:00:44.101544 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:44.101554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:44.101593 (+ 39us) server_negotiation.cc:407] Connection header received
0927 14:00:44.103068 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:44.103155 (+ 87us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:44.103627 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:44.103937 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:44.104577 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:44.105893 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:44.107786 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:44.108439 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:44.108563 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:44.109801 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:44.109810 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:44.109823 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:44.110367 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:44.110445 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:44.110468 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:44.110558 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:44.110786 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 14:00:44.111168 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":368,"threads_started":1}
I20260927 14:00:44.113478 8998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c: Bootstrap starting.
I20260927 14:00:44.116832 8998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:44.117785 8998 log.cc:824] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:44.120376 8998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c: No bootstrap required, opened a new log
I20260927 14:00:44.127516 8998 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ced684ddc24db8bf5b779f95a1a78c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33843 } }
I20260927 14:00:44.127969 8998 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:44.128074 8998 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ced684ddc24db8bf5b779f95a1a78c, State: Initialized, Role: FOLLOWER
I20260927 14:00:44.128615 8998 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ced684ddc24db8bf5b779f95a1a78c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33843 } }
I20260927 14:00:44.128846 8998 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:00:44.128991 8998 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:00:44.129176 8998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:44.131286 8998 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ced684ddc24db8bf5b779f95a1a78c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33843 } }
I20260927 14:00:44.131769 8998 leader_election.cc:302] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0ced684ddc24db8bf5b779f95a1a78c; no voters:
I20260927 14:00:44.132332 8998 leader_election.cc:288] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:00:44.132483 9003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:44.133208 9003 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [term 1 LEADER]: Becoming Leader. State: Replica: c0ced684ddc24db8bf5b779f95a1a78c, State: Running, Role: LEADER
I20260927 14:00:44.133724 9003 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ced684ddc24db8bf5b779f95a1a78c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33843 } }
I20260927 14:00:44.134344 8998 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:00:44.136546 9005 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0ced684ddc24db8bf5b779f95a1a78c. Latest consensus state: current_term: 1 leader_uuid: "c0ced684ddc24db8bf5b779f95a1a78c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ced684ddc24db8bf5b779f95a1a78c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33843 } } }
I20260927 14:00:44.137010 9005 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:44.137504 9004 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0ced684ddc24db8bf5b779f95a1a78c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ced684ddc24db8bf5b779f95a1a78c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33843 } } }
I20260927 14:00:44.137847 9004 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:44.139315 9012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:00:44.143687 9012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:00:44.151396 9012 catalog_manager.cc:1347] Generated new cluster ID: f8987fbfafed4a5294e8c04bccd07648
I20260927 14:00:44.151497 9012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:00:44.176927 9012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:00:44.178826 9012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:00:44.195865 9012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c: Generated new TSK 0
I20260927 14:00:44.196772 9012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:00:44.206620 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:33843
--builtin_ntp_servers=127.22.26.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 14:00:44.547673 9022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:44.547938 9022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:44.548108 9022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:44.555151 9022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:44.555243 9022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:44.555459 9022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 14:00:44.564545 9022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:33843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:44.565879 9022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:44.567749 9022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:44.580637 9027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:44.580601 9030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:44.580564 9028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:44.582194 9022 server_base.cc:1034] running on GCE node
I20260927 14:00:44.582919 9022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:44.584021 9022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:44.585242 9022 hybrid_clock.cc:648] HybridClock initialized: now 1790517644585190 us; error 38 us; skew 500 ppm
I20260927 14:00:44.585599 9022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 14:00:44.588227 9022 webserver.cc:460] Webserver started at http://127.22.26.193:43395/ using document root <none> and password file <none>
I20260927 14:00:44.589013 9022 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:44.589129 9022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:44.589458 9022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:44.592518 9022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "638981ecd2cd4a2e8b917fd962d4f9b1"
format_stamp: "Formatted at 2026-09-27 14:00:44 on dist-test-slave-gkks"
server_key: "8541900f2afa87cdf85957173a342cbb"
server_key_iv: "ca3784839024748e95855d8eeedb7f60"
server_key_version: "encryptionkey@0"
I20260927 14:00:44.593338 9022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "638981ecd2cd4a2e8b917fd962d4f9b1"
format_stamp: "Formatted at 2026-09-27 14:00:44 on dist-test-slave-gkks"
server_key: "8541900f2afa87cdf85957173a342cbb"
server_key_iv: "ca3784839024748e95855d8eeedb7f60"
server_key_version: "encryptionkey@0"
I20260927 14:00:44.598640 9022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 14:00:44.602574 9036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:44.604188 9022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:00:44.604382 9022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "638981ecd2cd4a2e8b917fd962d4f9b1"
format_stamp: "Formatted at 2026-09-27 14:00:44 on dist-test-slave-gkks"
server_key: "8541900f2afa87cdf85957173a342cbb"
server_key_iv: "ca3784839024748e95855d8eeedb7f60"
server_key_version: "encryptionkey@0"
I20260927 14:00:44.604610 9022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:44.629755 9022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:44.630640 9022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:44.630965 9022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:44.631994 9022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:44.633512 9022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:44.633637 9022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:44.633776 9022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:44.633868 9022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:44.669979 9022 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:46071
I20260927 14:00:44.670069 9148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:46071 every 8 connection(s)
I20260927 14:00:44.671439 9022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 14:00:44.675410 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 9022
I20260927 14:00:44.675587 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 14:00:44.676141 22635 external_mini_cluster.cc:1428] Setting key af6bba2500d0ade7d2737d3d101e0691
I20260927 14:00:44.679203 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:33843
--builtin_ntp_servers=127.22.26.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 14:00:44.685343 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:44.673872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:49927 (local address 127.22.26.254:33843)
0927 14:00:44.674339 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:44.674344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:44.675541 (+ 1197us) server_negotiation.cc:407] Connection header received
0927 14:00:44.677627 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:44.677633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:44.677710 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:44.677848 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:44.679220 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:44.680044 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:44.682517 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:44.682898 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:44.682988 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:44.684607 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:44.684612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:44.684615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:44.684633 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:44.684671 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:44.684683 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:44.684755 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:44.685012 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 14:00:44.685088 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":253,"threads_started":1}
I20260927 14:00:44.686733 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:44.674337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:33843 (local address 127.22.26.193:49927)
0927 14:00:44.675339 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:44.675396 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:44.677377 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:44.678071 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:44.678083 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:44.678208 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:44.678985 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:44.679011 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:44.680236 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:44.680245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:44.682368 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:44.682385 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:44.683158 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:44.683168 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:44.683491 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:44.684252 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:00:44.684257 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:44.684310 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:00:44.684415 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:44.684785 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:44.684798 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:44.684809 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 14:00:44.684981 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 14:00:44.685399 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":257,"threads_started":1}
I20260927 14:00:44.688352 9149 heartbeater.cc:347] Connected to a master server at 127.22.26.254:33843
I20260927 14:00:44.688685 9149 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:44.689584 9149 heartbeater.cc:511] Master 127.22.26.254:33843 requested a full tablet report, sending...
I20260927 14:00:44.691788 8963 ts_manager.cc:194] Registered new tserver with Master: 638981ecd2cd4a2e8b917fd962d4f9b1 (127.22.26.193:46071)
I20260927 14:00:44.693949 8963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.26.193:49927
W20260927 14:00:45.013283 9153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:45.013576 9153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:45.013669 9153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:45.020771 9153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:45.020874 9153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:45.020979 9153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 14:00:45.029942 9153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:33843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:45.031240 9153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:45.033008 9153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:45.046412 9161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:45.046608 9158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:45.046592 9159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:45.047724 9153 server_base.cc:1034] running on GCE node
I20260927 14:00:45.048362 9153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:45.049278 9153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:45.050485 9153 hybrid_clock.cc:648] HybridClock initialized: now 1790517645050433 us; error 42 us; skew 500 ppm
I20260927 14:00:45.050813 9153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 14:00:45.053099 9153 webserver.cc:460] Webserver started at http://127.22.26.194:37805/ using document root <none> and password file <none>
I20260927 14:00:45.053791 9153 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:45.053905 9153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:45.054229 9153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:45.057080 9153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "0fa3b58931c944c1b84a4fd3692f4e10"
format_stamp: "Formatted at 2026-09-27 14:00:45 on dist-test-slave-gkks"
server_key: "52861480dabad8d3bc1b4a240d98677a"
server_key_iv: "e29e15372492e9b245cf6e42ce907bd8"
server_key_version: "encryptionkey@0"
I20260927 14:00:45.057855 9153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "0fa3b58931c944c1b84a4fd3692f4e10"
format_stamp: "Formatted at 2026-09-27 14:00:45 on dist-test-slave-gkks"
server_key: "52861480dabad8d3bc1b4a240d98677a"
server_key_iv: "e29e15372492e9b245cf6e42ce907bd8"
server_key_version: "encryptionkey@0"
I20260927 14:00:45.063019 9153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 14:00:45.066588 9167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:45.067981 9153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:00:45.068151 9153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "0fa3b58931c944c1b84a4fd3692f4e10"
format_stamp: "Formatted at 2026-09-27 14:00:45 on dist-test-slave-gkks"
server_key: "52861480dabad8d3bc1b4a240d98677a"
server_key_iv: "e29e15372492e9b245cf6e42ce907bd8"
server_key_version: "encryptionkey@0"
I20260927 14:00:45.068378 9153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:45.085325 9153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:45.086089 9153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:45.086390 9153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:45.087441 9153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:45.088938 9153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:45.089037 9153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:45.089148 9153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:45.089219 9153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:45.124554 9153 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:34029
I20260927 14:00:45.124642 9279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:34029 every 8 connection(s)
I20260927 14:00:45.125912 9153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 14:00:45.127560 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 9153
I20260927 14:00:45.127743 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 14:00:45.128190 22635 external_mini_cluster.cc:1428] Setting key 78ac3eaaf090f2f99631600e27b24d50
I20260927 14:00:45.131505 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:33843
--builtin_ntp_servers=127.22.26.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 14:00:45.139639 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.128727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:50387 (local address 127.22.26.254:33843)
0927 14:00:45.128953 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:45.128960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:45.130776 (+ 1816us) server_negotiation.cc:407] Connection header received
0927 14:00:45.132726 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:45.132731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:45.132818 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:45.132941 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:45.134028 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.134916 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.136655 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.137051 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.137191 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.138586 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:45.138594 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:45.138601 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:45.138627 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:45.138674 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:45.138692 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:45.138809 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:45.139147 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 14:00:45.139271 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 14:00:45.140956 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.129492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:33843 (local address 127.22.26.194:50387)
0927 14:00:45.130620 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:45.130658 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:45.132459 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:45.133180 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:45.133188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:45.133282 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:45.133859 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.133876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.135069 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.135127 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.136509 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.136521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.137285 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.137292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.137519 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.138271 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:00:45.138274 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:45.138312 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:00:45.138393 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:45.138838 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:45.138848 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:45.138855 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 14:00:45.139143 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 14:00:45.139701 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":327,"threads_started":1}
I20260927 14:00:45.142414 9280 heartbeater.cc:347] Connected to a master server at 127.22.26.254:33843
I20260927 14:00:45.142724 9280 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:45.143554 9280 heartbeater.cc:511] Master 127.22.26.254:33843 requested a full tablet report, sending...
I20260927 14:00:45.144989 8963 ts_manager.cc:194] Registered new tserver with Master: 0fa3b58931c944c1b84a4fd3692f4e10 (127.22.26.194:34029)
I20260927 14:00:45.145854 8963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.26.194:50387
W20260927 14:00:45.473537 9283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:45.473856 9283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:45.474010 9283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:45.481185 9283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:45.481317 9283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:45.481487 9283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 14:00:45.490641 9283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:33843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:45.492034 9283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:45.493932 9283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:45.507531 9288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:45.507469 9291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:45.508316 9289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:45.508333 9283 server_base.cc:1034] running on GCE node
I20260927 14:00:45.509100 9283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:45.510134 9283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:45.511359 9283 hybrid_clock.cc:648] HybridClock initialized: now 1790517645511309 us; error 66 us; skew 500 ppm
I20260927 14:00:45.511690 9283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 14:00:45.513950 9283 webserver.cc:460] Webserver started at http://127.22.26.195:34319/ using document root <none> and password file <none>
I20260927 14:00:45.514657 9283 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:45.514801 9283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:45.515169 9283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:45.518013 9283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "9090e042d19f484698ebff6e43f56c5f"
format_stamp: "Formatted at 2026-09-27 14:00:45 on dist-test-slave-gkks"
server_key: "fb64f2c9a18dda9bdf11b2cdd1bbe534"
server_key_iv: "28108bb3ba1e2016c2db1f18f646e696"
server_key_version: "encryptionkey@0"
I20260927 14:00:45.518796 9283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "9090e042d19f484698ebff6e43f56c5f"
format_stamp: "Formatted at 2026-09-27 14:00:45 on dist-test-slave-gkks"
server_key: "fb64f2c9a18dda9bdf11b2cdd1bbe534"
server_key_iv: "28108bb3ba1e2016c2db1f18f646e696"
server_key_version: "encryptionkey@0"
I20260927 14:00:45.524030 9283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 14:00:45.527578 9297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:45.529006 9283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 14:00:45.529191 9283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "9090e042d19f484698ebff6e43f56c5f"
format_stamp: "Formatted at 2026-09-27 14:00:45 on dist-test-slave-gkks"
server_key: "fb64f2c9a18dda9bdf11b2cdd1bbe534"
server_key_iv: "28108bb3ba1e2016c2db1f18f646e696"
server_key_version: "encryptionkey@0"
I20260927 14:00:45.529412 9283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:45.547236 9283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:45.548022 9283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:45.548332 9283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:45.549260 9283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:45.550670 9283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:45.550807 9283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:45.550933 9283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:45.551020 9283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:45.583976 9283 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:36783
I20260927 14:00:45.584019 9409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:36783 every 8 connection(s)
I20260927 14:00:45.585297 9283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 14:00:45.587440 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 9283
I20260927 14:00:45.587607 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 14:00:45.588148 22635 external_mini_cluster.cc:1428] Setting key d14ed8e38ba7f0b1f53b98e7fb91cf1e
I20260927 14:00:45.597991 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.587850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:59527 (local address 127.22.26.254:33843)
0927 14:00:45.588065 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:45.588071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:45.589782 (+ 1711us) server_negotiation.cc:407] Connection header received
0927 14:00:45.591691 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:45.591696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:45.591779 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:45.591888 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:45.592986 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.593722 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.595245 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.595656 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.595787 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.597171 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:45.597176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:45.597179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:45.597201 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:45.597232 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:45.597243 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:45.597330 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:45.597548 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 14:00:45.597647 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 14:00:45.598884 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.588509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:33843 (local address 127.22.26.195:59527)
0927 14:00:45.589620 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:45.589677 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:45.591449 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:45.592107 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:45.592114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:45.592195 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:45.592794 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.592833 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.593858 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.593863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.595053 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.595064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.595868 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.595875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.596099 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.596875 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:00:45.596877 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:45.596914 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:00:45.596982 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:45.597344 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:45.597356 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:45.597371 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 14:00:45.597536 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 14:00:45.597905 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":335,"threads_started":1}
I20260927 14:00:45.600121 9410 heartbeater.cc:347] Connected to a master server at 127.22.26.254:33843
I20260927 14:00:45.600365 9410 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:45.601024 9410 heartbeater.cc:511] Master 127.22.26.254:33843 requested a full tablet report, sending...
I20260927 14:00:45.602438 8963 ts_manager.cc:194] Registered new tserver with Master: 9090e042d19f484698ebff6e43f56c5f (127.22.26.195:36783)
I20260927 14:00:45.603449 8963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.26.195:59527
I20260927 14:00:45.603477 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:00:45.613586 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.607595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.22.26.254:33843)
0927 14:00:45.607722 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:45.607727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:45.608292 (+ 565us) server_negotiation.cc:407] Connection header received
0927 14:00:45.608400 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:45.608404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:45.608459 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:45.608537 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:45.609012 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.609678 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.611033 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.611542 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.611641 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.612865 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:45.612870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:45.612874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:45.612892 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:45.612925 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:45.612938 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:45.613092 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:45.613340 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 14:00:45.613385 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 14:00:45.617581 8963 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:00:45.624207 8963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:00:45.652089 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.636415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:36783 (local address 127.0.0.1:39518)
0927 14:00:45.637198 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:45.637251 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:45.637468 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:45.641387 (+ 3919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:45.641394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:45.641424 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:45.641883 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.641902 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.644503 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.644512 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.647676 (+ 3164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.647697 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.650716 (+ 3019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.650725 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.651046 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.651161 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 14:00:45.651338 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 14:00:45.651615 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":342,"threads_started":1}
I20260927 14:00:45.653381 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.636734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.22.26.195:36783)
0927 14:00:45.640426 (+ 3692us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:45.640433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:45.640460 (+ 27us) server_negotiation.cc:407] Connection header received
0927 14:00:45.640535 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:45.640541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:45.640814 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:45.641010 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:45.642181 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.644300 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.647951 (+ 3651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.649997 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.650175 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.650556 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:45.652620 (+ 2064us) server_negotiation.cc:299] Negotiation successful
0927 14:00:45.652867 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3518,"thread_start_us":3330,"threads_started":1}
I20260927 14:00:45.657759 9345 tablet_service.cc:1467] Processing CreateTablet for tablet 1aaf65d661484d42b50358577f7fd6c9 (DEFAULT_TABLE table=test-table [id=82cfe4a2b4164ea28f43470cbfaf20cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:45.659421 9345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aaf65d661484d42b50358577f7fd6c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:45.659792 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.636563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:46071 (local address 127.0.0.1:33476)
0927 14:00:45.639602 (+ 3039us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:45.639626 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:45.639771 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:45.650876 (+ 11105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:45.650881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:45.650909 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:45.651390 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.651403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.654567 (+ 3164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.654573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.657028 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.657046 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.658916 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.658922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.659117 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.659150 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 14:00:45.659243 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 14:00:45.659372 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2849,"thread_start_us":2691,"threads_started":1}
I20260927 14:00:45.659981 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.637151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.22.26.193:46071)
0927 14:00:45.649825 (+ 12674us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:45.649838 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:45.649880 (+ 42us) server_negotiation.cc:407] Connection header received
0927 14:00:45.650017 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:45.650030 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:45.650359 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:45.651601 (+ 1242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:45.651848 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.654373 (+ 2525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.657294 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.658703 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.658885 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.659147 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:45.659467 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 14:00:45.659633 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12432,"thread_start_us":13412,"threads_started":1}
I20260927 14:00:45.663482 9084 tablet_service.cc:1467] Processing CreateTablet for tablet 1aaf65d661484d42b50358577f7fd6c9 (DEFAULT_TABLE table=test-table [id=82cfe4a2b4164ea28f43470cbfaf20cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:45.664827 9084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aaf65d661484d42b50358577f7fd6c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:45.665079 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.640951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:34029 (local address 127.0.0.1:50100)
0927 14:00:45.656357 (+ 15406us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:45.656376 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:45.656493 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:45.658060 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:45.658066 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:45.658091 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:45.658397 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.658408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.660284 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.660289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.662343 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.662358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.664241 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.664248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.664423 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.664465 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 14:00:45.664561 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 14:00:45.664689 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15270,"thread_start_us":15623,"threads_started":1}
I20260927 14:00:45.665181 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.641431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.22.26.194:34029)
0927 14:00:45.657460 (+ 16029us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:45.657469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:45.657497 (+ 28us) server_negotiation.cc:407] Connection header received
0927 14:00:45.657592 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:45.657599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:45.657841 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:45.658038 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:45.658611 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.660110 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.662590 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.664017 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.664150 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.664371 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:45.664622 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 14:00:45.664802 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15838,"thread_start_us":16755,"threads_started":1}
I20260927 14:00:45.669371 9215 tablet_service.cc:1467] Processing CreateTablet for tablet 1aaf65d661484d42b50358577f7fd6c9 (DEFAULT_TABLE table=test-table [id=82cfe4a2b4164ea28f43470cbfaf20cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:45.670717 9215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aaf65d661484d42b50358577f7fd6c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:45.683647 9430 tablet_bootstrap.cc:492] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10: Bootstrap starting.
I20260927 14:00:45.685618 9428 tablet_bootstrap.cc:492] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1: Bootstrap starting.
I20260927 14:00:45.687395 9429 tablet_bootstrap.cc:492] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f: Bootstrap starting.
I20260927 14:00:45.687425 9430 tablet_bootstrap.cc:654] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:45.689100 9430 log.cc:824] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:45.689219 9428 tablet_bootstrap.cc:654] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:45.690596 9428 log.cc:824] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:45.691020 9429 tablet_bootstrap.cc:654] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:45.692260 9429 log.cc:824] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:45.694083 9428 tablet_bootstrap.cc:492] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1: No bootstrap required, opened a new log
I20260927 14:00:45.694356 9428 ts_tablet_manager.cc:1397] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 14:00:45.695269 9430 tablet_bootstrap.cc:492] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10: No bootstrap required, opened a new log
I20260927 14:00:45.695331 9429 tablet_bootstrap.cc:492] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f: No bootstrap required, opened a new log
I20260927 14:00:45.695536 9430 ts_tablet_manager.cc:1397] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20260927 14:00:45.695582 9429 ts_tablet_manager.cc:1397] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 14:00:45.697124 9149 heartbeater.cc:503] Master 127.22.26.254:33843 was elected leader, sending a full tablet report...
I20260927 14:00:45.704586 9430 raft_consensus.cc:348] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9090e042d19f484698ebff6e43f56c5f" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36783 } } peers { permanent_uuid: "638981ecd2cd4a2e8b917fd962d4f9b1" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46071 } } peers { permanent_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 34029 } }
I20260927 14:00:45.705003 9430 raft_consensus.cc:374] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:45.705094 9430 raft_consensus.cc:729] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa3b58931c944c1b84a4fd3692f4e10, State: Initialized, Role: FOLLOWER
I20260927 14:00:45.705808 9430 consensus_queue.cc:260] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9090e042d19f484698ebff6e43f56c5f" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36783 } } peers { permanent_uuid: "638981ecd2cd4a2e8b917fd962d4f9b1" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46071 } } peers { permanent_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 34029 } }
I20260927 14:00:45.707050 9280 heartbeater.cc:503] Master 127.22.26.254:33843 was elected leader, sending a full tablet report...
I20260927 14:00:45.706867 9429 raft_consensus.cc:348] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9090e042d19f484698ebff6e43f56c5f" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36783 } } peers { permanent_uuid: "638981ecd2cd4a2e8b917fd962d4f9b1" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46071 } } peers { permanent_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 34029 } }
I20260927 14:00:45.707332 9429 raft_consensus.cc:374] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:45.707451 9429 raft_consensus.cc:729] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9090e042d19f484698ebff6e43f56c5f, State: Initialized, Role: FOLLOWER
I20260927 14:00:45.707906 9430 ts_tablet_manager.cc:1428] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260927 14:00:45.708016 9429 consensus_queue.cc:260] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9090e042d19f484698ebff6e43f56c5f" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36783 } } peers { permanent_uuid: "638981ecd2cd4a2e8b917fd962d4f9b1" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46071 } } peers { permanent_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 34029 } }
I20260927 14:00:45.709146 9410 heartbeater.cc:503] Master 127.22.26.254:33843 was elected leader, sending a full tablet report...
I20260927 14:00:45.709740 9429 ts_tablet_manager.cc:1428] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 14:00:45.708460 9428 raft_consensus.cc:348] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9090e042d19f484698ebff6e43f56c5f" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36783 } } peers { permanent_uuid: "638981ecd2cd4a2e8b917fd962d4f9b1" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46071 } } peers { permanent_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 34029 } }
I20260927 14:00:45.710467 9428 raft_consensus.cc:374] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:45.710577 9428 raft_consensus.cc:729] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638981ecd2cd4a2e8b917fd962d4f9b1, State: Initialized, Role: FOLLOWER
I20260927 14:00:45.711227 9428 consensus_queue.cc:260] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9090e042d19f484698ebff6e43f56c5f" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36783 } } peers { permanent_uuid: "638981ecd2cd4a2e8b917fd962d4f9b1" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46071 } } peers { permanent_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 34029 } }
I20260927 14:00:45.712952 9428 ts_tablet_manager.cc:1428] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1: Time spent starting tablet: real 0.018s user 0.014s sys 0.001s
I20260927 14:00:45.718302 9434 raft_consensus.cc:482] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:00:45.718614 9434 raft_consensus.cc:504] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9090e042d19f484698ebff6e43f56c5f" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36783 } } peers { permanent_uuid: "638981ecd2cd4a2e8b917fd962d4f9b1" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46071 } } peers { permanent_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 34029 } }
I20260927 14:00:45.720288 9434 leader_election.cc:288] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9090e042d19f484698ebff6e43f56c5f (127.22.26.195:36783), 638981ecd2cd4a2e8b917fd962d4f9b1 (127.22.26.193:46071)
I20260927 14:00:45.726974 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.720984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:52957 (local address 127.22.26.193:46071)
0927 14:00:45.721190 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:45.721196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:45.721539 (+ 343us) server_negotiation.cc:407] Connection header received
0927 14:00:45.721721 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:45.721726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:45.721812 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:45.721913 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:45.722516 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.723457 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.725256 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.725994 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.726171 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.726252 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:45.726566 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 14:00:45.726696 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 14:00:45.726980 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.720503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:51781 (local address 127.22.26.195:36783)
0927 14:00:45.720711 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:45.720720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:45.721276 (+ 556us) server_negotiation.cc:407] Connection header received
0927 14:00:45.721494 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:45.721502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:45.721582 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:45.721667 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:45.722311 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.723164 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.725092 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.726008 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.726204 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.726274 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:45.726614 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 14:00:45.726746 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 14:00:45.727340 9437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.720448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:36783 (local address 127.22.26.194:51781)
0927 14:00:45.721165 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:45.721194 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:45.721350 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:45.721718 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:45.721723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:45.721750 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:45.722165 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.722175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.723285 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.723290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.724898 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.724910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.726224 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.726230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.726437 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.726460 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 14:00:45.726569 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 14:00:45.726696 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":41,"thread_start_us":330,"threads_started":1}
I20260927 14:00:45.727416 9438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.720724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:46071 (local address 127.22.26.194:52957)
0927 14:00:45.721434 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:45.721450 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:45.721574 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:45.721940 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:45.721944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:45.721963 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:45.722358 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.722375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.723601 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.723605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.725068 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:45.725083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.726194 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.726205 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:45.726410 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.726436 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 14:00:45.726508 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 14:00:45.726657 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":3,"thread_start_us":371,"threads_started":1}
I20260927 14:00:45.728478 9104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aaf65d661484d42b50358577f7fd6c9" candidate_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638981ecd2cd4a2e8b917fd962d4f9b1" is_pre_election: true
I20260927 14:00:45.728484 9365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aaf65d661484d42b50358577f7fd6c9" candidate_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9090e042d19f484698ebff6e43f56c5f" is_pre_election: true
I20260927 14:00:45.729067 9365 raft_consensus.cc:2454] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa3b58931c944c1b84a4fd3692f4e10 in term 0.
I20260927 14:00:45.729069 9104 raft_consensus.cc:2454] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa3b58931c944c1b84a4fd3692f4e10 in term 0.
I20260927 14:00:45.729871 9171 leader_election.cc:302] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0fa3b58931c944c1b84a4fd3692f4e10, 638981ecd2cd4a2e8b917fd962d4f9b1, 9090e042d19f484698ebff6e43f56c5f; no voters:
I20260927 14:00:45.730316 9434 raft_consensus.cc:2781] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:00:45.730489 9434 raft_consensus.cc:482] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:00:45.730612 9434 raft_consensus.cc:3037] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:45.732878 9434 raft_consensus.cc:504] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9090e042d19f484698ebff6e43f56c5f" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36783 } } peers { permanent_uuid: "638981ecd2cd4a2e8b917fd962d4f9b1" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46071 } } peers { permanent_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 34029 } }
I20260927 14:00:45.733479 9434 leader_election.cc:288] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [CANDIDATE]: Term 1 election: Requested vote from peers 9090e042d19f484698ebff6e43f56c5f (127.22.26.195:36783), 638981ecd2cd4a2e8b917fd962d4f9b1 (127.22.26.193:46071)
I20260927 14:00:45.733942 9104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aaf65d661484d42b50358577f7fd6c9" candidate_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638981ecd2cd4a2e8b917fd962d4f9b1"
I20260927 14:00:45.733934 9365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aaf65d661484d42b50358577f7fd6c9" candidate_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9090e042d19f484698ebff6e43f56c5f"
I20260927 14:00:45.734181 9104 raft_consensus.cc:3037] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:45.734222 9365 raft_consensus.cc:3037] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:45.736668 9104 raft_consensus.cc:2454] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fa3b58931c944c1b84a4fd3692f4e10 in term 1.
I20260927 14:00:45.736713 9365 raft_consensus.cc:2454] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fa3b58931c944c1b84a4fd3692f4e10 in term 1.
I20260927 14:00:45.737092 9169 leader_election.cc:302] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa3b58931c944c1b84a4fd3692f4e10, 638981ecd2cd4a2e8b917fd962d4f9b1; no voters:
I20260927 14:00:45.737481 9434 raft_consensus.cc:2781] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:45.737959 9434 raft_consensus.cc:686] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [term 1 LEADER]: Becoming Leader. State: Replica: 0fa3b58931c944c1b84a4fd3692f4e10, State: Running, Role: LEADER
I20260927 14:00:45.738494 9434 consensus_queue.cc:237] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9090e042d19f484698ebff6e43f56c5f" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36783 } } peers { permanent_uuid: "638981ecd2cd4a2e8b917fd962d4f9b1" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46071 } } peers { permanent_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 34029 } }
I20260927 14:00:45.743609 8963 catalog_manager.cc:5495] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fa3b58931c944c1b84a4fd3692f4e10 (127.22.26.194). New cstate: current_term: 1 leader_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9090e042d19f484698ebff6e43f56c5f" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "638981ecd2cd4a2e8b917fd962d4f9b1" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fa3b58931c944c1b84a4fd3692f4e10" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 34029 } health_report { overall_health: HEALTHY } } }
I20260927 14:00:45.775692 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.769899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.22.26.194:34029)
0927 14:00:45.770110 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:45.770116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:45.770218 (+ 102us) server_negotiation.cc:407] Connection header received
0927 14:00:45.770471 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:45.770475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:45.770573 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:45.770736 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:00:45.771167 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.771987 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.773398 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.773820 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.773962 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.774564 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:00:45.775060 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:00:45.775274 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:45.775408 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 14:00:45.775450 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:00:45.783716 9365 raft_consensus.cc:1261] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f [term 1 FOLLOWER]: Refusing update from remote peer 0fa3b58931c944c1b84a4fd3692f4e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:00:45.783716 9104 raft_consensus.cc:1261] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1 [term 1 FOLLOWER]: Refusing update from remote peer 0fa3b58931c944c1b84a4fd3692f4e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:00:45.784863 9434 consensus_queue.cc:1035] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "638981ecd2cd4a2e8b917fd962d4f9b1" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:00:45.785192 9439 consensus_queue.cc:1035] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9090e042d19f484698ebff6e43f56c5f" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:00:45.795998 9442 mvcc.cc:205] Tried to move back new op lower bound from 7333960277119868928 to 7333960276949553152. Current Snapshot: MvccSnapshot[applied={T|T < 7333960277119868928}]
I20260927 14:00:45.800604 9443 mvcc.cc:205] Tried to move back new op lower bound from 7333960277119868928 to 7333960276949553152. Current Snapshot: MvccSnapshot[applied={T|T < 7333960277119868928}]
I20260927 14:00:45.809517 8963 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:00:45.809799 8963 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:00:45.813457 8963 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0ced684ddc24db8bf5b779f95a1a78c: Sending DeleteTablet for 3 replicas of tablet 1aaf65d661484d42b50358577f7fd6c9
I20260927 14:00:45.814577 9084 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aaf65d661484d42b50358577f7fd6c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:45 UTC) from {username='slave'} at 127.0.0.1:33476
I20260927 14:00:45.814694 9345 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aaf65d661484d42b50358577f7fd6c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:45 UTC) from {username='slave'} at 127.0.0.1:39518
I20260927 14:00:45.815524 9452 tablet_replica.cc:330] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1: stopping tablet replica
I20260927 14:00:45.815790 9215 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aaf65d661484d42b50358577f7fd6c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:45 UTC) from {username='slave'} at 127.0.0.1:50100
I20260927 14:00:45.815961 9452 raft_consensus.cc:2229] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:00:45.816444 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 9022
I20260927 14:00:45.816475 9452 raft_consensus.cc:2258] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:45.816738 9454 tablet_replica.cc:330] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10: stopping tablet replica
I20260927 14:00:45.817626 9454 raft_consensus.cc:2229] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:00:45.818320 9454 raft_consensus.cc:2258] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:45.818430 9452 ts_tablet_manager.cc:1905] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:45.820425 9454 ts_tablet_manager.cc:1905] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:45.825470 9452 ts_tablet_manager.cc:1918] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:45.825650 9452 log.cc:1196] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/1aaf65d661484d42b50358577f7fd6c9
I20260927 14:00:45.826284 9452 ts_tablet_manager.cc:1939] T 1aaf65d661484d42b50358577f7fd6c9 P 638981ecd2cd4a2e8b917fd962d4f9b1: Deleting consensus metadata
I20260927 14:00:45.829272 9454 ts_tablet_manager.cc:1918] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:45.829300 8948 catalog_manager.cc:4841] TS 638981ecd2cd4a2e8b917fd962d4f9b1 (127.22.26.193:46071): tablet 1aaf65d661484d42b50358577f7fd6c9 (table test-table [id=82cfe4a2b4164ea28f43470cbfaf20cb]) successfully deleted
I20260927 14:00:45.829547 9454 log.cc:1196] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/1aaf65d661484d42b50358577f7fd6c9
I20260927 14:00:45.830145 9454 ts_tablet_manager.cc:1939] T 1aaf65d661484d42b50358577f7fd6c9 P 0fa3b58931c944c1b84a4fd3692f4e10: Deleting consensus metadata
I20260927 14:00:45.831823 8947 catalog_manager.cc:4841] TS 0fa3b58931c944c1b84a4fd3692f4e10 (127.22.26.194:34029): tablet 1aaf65d661484d42b50358577f7fd6c9 (table test-table [id=82cfe4a2b4164ea28f43470cbfaf20cb]) successfully deleted
I20260927 14:00:45.833370 9453 tablet_replica.cc:330] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f: stopping tablet replica
I20260927 14:00:45.833863 9453 raft_consensus.cc:2229] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:00:45.834359 9453 raft_consensus.cc:2258] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:45.835772 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.818607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.22.26.193:46071)
0927 14:00:45.818779 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:45.818786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:45.827608 (+ 8822us) server_negotiation.cc:407] Connection header received
0927 14:00:45.827834 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:45.827840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:45.827975 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:45.828090 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:45.828729 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.829880 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.832081 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.832570 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.832707 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.834022 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:45.834033 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:45.834051 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:45.834897 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:45.834957 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:45.834991 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:45.835131 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:45.835295 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 14:00:45.835413 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:00:45.836484 9144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:45.836828 9453 ts_tablet_manager.cc:1905] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:45.846251 9453 ts_tablet_manager.cc:1918] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:45.846485 9453 log.cc:1196] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/1aaf65d661484d42b50358577f7fd6c9
I20260927 14:00:45.847126 9453 ts_tablet_manager.cc:1939] T 1aaf65d661484d42b50358577f7fd6c9 P 9090e042d19f484698ebff6e43f56c5f: Deleting consensus metadata
I20260927 14:00:45.850564 8950 catalog_manager.cc:4841] TS 9090e042d19f484698ebff6e43f56c5f (127.22.26.195:36783): tablet 1aaf65d661484d42b50358577f7fd6c9 (table test-table [id=82cfe4a2b4164ea28f43470cbfaf20cb]) successfully deleted
I20260927 14:00:45.936612 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 9022
I20260927 14:00:45.954038 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 9153
I20260927 14:00:45.961185 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:45.954857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.22.26.194:34029)
0927 14:00:45.955031 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:45.955036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:45.955140 (+ 104us) server_negotiation.cc:407] Connection header received
0927 14:00:45.955376 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:45.955381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:45.955494 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:45.955614 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:45.956024 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.956814 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.958073 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:45.958520 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:45.958658 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:45.959779 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:45.959788 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:45.959802 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:45.960421 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:45.960470 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:45.960496 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:45.960612 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:45.960745 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 14:00:45.960851 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:00:45.961810 9275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:46.074541 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 9153
I20260927 14:00:46.093302 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 9283
I20260927 14:00:46.100731 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:46.094292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.22.26.195:36783)
0927 14:00:46.094510 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:46.094516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:46.094537 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:00:46.094698 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:46.094704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:46.094810 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:46.094930 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:46.095426 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:46.096210 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:46.097577 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:46.098057 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:46.098201 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:46.099397 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:46.099407 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:46.099425 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:46.100026 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:46.100073 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:46.100093 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:46.100222 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:46.100329 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 14:00:46.100437 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 14:00:46.101527 9405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:46.191833 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 9283
I20260927 14:00:46.208496 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 8932
I20260927 14:00:46.209656 8993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:46.322012 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 8932
2026-09-27T14:00:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2721 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:00:46 dist-test-slave-gkks krb5kdc[9462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:00:46 dist-test-slave-gkks krb5kdc[9462](info): set up 2 sockets
Sep 27 14:00:46 dist-test-slave-gkks krb5kdc[9462](info): commencing operation
krb5kdc: starting...
W20260927 14:00:47.447624 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:00:47 dist-test-slave-gkks krb5kdc[9462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:00:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:00:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 14:00:47.700538 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:37861
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:37861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:48.032585 9475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:48.032894 9475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:48.033115 9475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:48.040439 9475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:00:48.040542 9475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:48.040650 9475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:48.040757 9475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:00:48.040848 9475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:00:48.050642 9475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:37861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:37861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:48.052027 9475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:48.053416 9475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:48.065683 9480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:48.065739 9483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:48.065642 9481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:48.066782 9475 server_base.cc:1034] running on GCE node
I20260927 14:00:48.067655 9475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:48.069062 9475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:48.070304 9475 hybrid_clock.cc:648] HybridClock initialized: now 1790517648070261 us; error 50 us; skew 500 ppm
I20260927 14:00:48.070660 9475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:48 dist-test-slave-gkks krb5kdc[9462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517648, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:48.080513 9475 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 14:00:48.082093 9475 webserver.cc:460] Webserver started at http://127.22.26.254:38435/ using document root <none> and password file <none>
I20260927 14:00:48.082799 9475 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:48.082942 9475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:48.083325 9475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:48.086303 9475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "fbf4d59369384bb89ae40a2b7f452bab"
format_stamp: "Formatted at 2026-09-27 14:00:48 on dist-test-slave-gkks"
server_key: "e62d8c0e632de98538e99ae6efc644a2"
server_key_iv: "89bb86ceb19858ee65e91b55d8f21c2e"
server_key_version: "encryptionkey@0"
I20260927 14:00:48.087052 9475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "fbf4d59369384bb89ae40a2b7f452bab"
format_stamp: "Formatted at 2026-09-27 14:00:48 on dist-test-slave-gkks"
server_key: "e62d8c0e632de98538e99ae6efc644a2"
server_key_iv: "89bb86ceb19858ee65e91b55d8f21c2e"
server_key_version: "encryptionkey@0"
I20260927 14:00:48.092289 9475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 14:00:48.095599 9490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:48.096827 9475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 14:00:48.097012 9475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "fbf4d59369384bb89ae40a2b7f452bab"
format_stamp: "Formatted at 2026-09-27 14:00:48 on dist-test-slave-gkks"
server_key: "e62d8c0e632de98538e99ae6efc644a2"
server_key_iv: "89bb86ceb19858ee65e91b55d8f21c2e"
server_key_version: "encryptionkey@0"
I20260927 14:00:48.097208 9475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:48.119764 9475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:48.124130 9475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:48.124428 9475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:48.143285 9475 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:37861
I20260927 14:00:48.143249 9541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:37861 every 8 connection(s)
I20260927 14:00:48.144647 9475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:00:48.148524 9542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:48.154732 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 9475
I20260927 14:00:48.154939 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:00:48.155555 22635 external_mini_cluster.cc:1428] Setting key cc07a6244907c3af12c3b0ccc5ec6e88
I20260927 14:00:48.159518 9542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab: Bootstrap starting.
I20260927 14:00:48.163362 9542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:48.164403 9542 log.cc:824] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:48.167622 9542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab: No bootstrap required, opened a new log
Sep 27 14:00:48 dist-test-slave-gkks krb5kdc[9462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:00:48.176596 9542 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf4d59369384bb89ae40a2b7f452bab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37861 } }
I20260927 14:00:48.176980 9542 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:48.177094 9542 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf4d59369384bb89ae40a2b7f452bab, State: Initialized, Role: FOLLOWER
I20260927 14:00:48.177759 9542 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf4d59369384bb89ae40a2b7f452bab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37861 } }
I20260927 14:00:48.178061 9542 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:00:48.178243 9542 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:00:48.178440 9542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:48.180004 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:48.157474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.22.26.254:37861)
0927 14:00:48.158227 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:48.158239 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:48.158278 (+ 39us) server_negotiation.cc:407] Connection header received
0927 14:00:48.160087 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:48.160116 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:48.160484 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:48.160807 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:48.162878 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:48.164382 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:48.166036 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:48.166746 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:48.166885 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:48.172090 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:48.172098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:48.172113 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:48.172157 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:48.175820 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:48.176501 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:48.176513 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:48.176520 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:48.176686 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:48.177156 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:48.177160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:48.177163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:48.177734 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:48.178099 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:48.178457 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 14:00:48.178884 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":366,"threads_started":1}
I20260927 14:00:48.181442 9542 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf4d59369384bb89ae40a2b7f452bab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37861 } }
I20260927 14:00:48.181957 9542 leader_election.cc:302] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbf4d59369384bb89ae40a2b7f452bab; no voters:
I20260927 14:00:48.182616 9542 leader_election.cc:288] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:00:48.182756 9547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:48.183873 9547 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [term 1 LEADER]: Becoming Leader. State: Replica: fbf4d59369384bb89ae40a2b7f452bab, State: Running, Role: LEADER
I20260927 14:00:48.184420 9547 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf4d59369384bb89ae40a2b7f452bab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37861 } }
I20260927 14:00:48.184983 9542 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:00:48.187263 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbf4d59369384bb89ae40a2b7f452bab. Latest consensus state: current_term: 1 leader_uuid: "fbf4d59369384bb89ae40a2b7f452bab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf4d59369384bb89ae40a2b7f452bab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37861 } } }
I20260927 14:00:48.187381 9548 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbf4d59369384bb89ae40a2b7f452bab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf4d59369384bb89ae40a2b7f452bab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37861 } } }
I20260927 14:00:48.187776 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:48.187808 9548 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:48.190011 9554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:00:48.195593 9554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:00:48.204521 9554 catalog_manager.cc:1347] Generated new cluster ID: 611deca6267e4d88ad92e9bc9b7d96e4
I20260927 14:00:48.204712 9554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:00:48.219558 9554 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:00:48.221130 9554 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:00:48.230846 9554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab: Generated new TSK 0
I20260927 14:00:48.231715 9554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
Entry for principal kudu/127.22.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab.
I20260927 14:00:48.304258 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:37861
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:48.643508 9568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:48.643780 9568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:48.643875 9568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:48.650863 9568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:48.650961 9568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:48.651073 9568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 14:00:48.660676 9568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.193.keytab
--principal=kudu/127.22.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:37861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:48.661942 9568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:48.663321 9568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:48.676040 9573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:48.676060 9576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:48.676093 9574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:48.677222 9568 server_base.cc:1034] running on GCE node
I20260927 14:00:48.677790 9568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:48.678683 9568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:48.679946 9568 hybrid_clock.cc:648] HybridClock initialized: now 1790517648679901 us; error 46 us; skew 500 ppm
I20260927 14:00:48.680377 9568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:48 dist-test-slave-gkks krb5kdc[9462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517648, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:48.683738 9568 init.cc:377] Logged in from keytab as kudu/127.22.26.193@KRBTEST.COM (short username kudu)
I20260927 14:00:48.685078 9568 webserver.cc:460] Webserver started at http://127.22.26.193:41221/ using document root <none> and password file <none>
I20260927 14:00:48.685722 9568 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:48.685822 9568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:48.686069 9568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:48.688975 9568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "2786b0334aaa4fb089993c1fcb0fe58c"
format_stamp: "Formatted at 2026-09-27 14:00:48 on dist-test-slave-gkks"
server_key: "4e410467c74319f50133c40d6f05f8c8"
server_key_iv: "6ee4e74ce7d24049c45863d3704164b0"
server_key_version: "encryptionkey@0"
I20260927 14:00:48.689750 9568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "2786b0334aaa4fb089993c1fcb0fe58c"
format_stamp: "Formatted at 2026-09-27 14:00:48 on dist-test-slave-gkks"
server_key: "4e410467c74319f50133c40d6f05f8c8"
server_key_iv: "6ee4e74ce7d24049c45863d3704164b0"
server_key_version: "encryptionkey@0"
I20260927 14:00:48.694938 9568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 14:00:48.698226 9583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:48.699612 9568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:00:48.699805 9568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "2786b0334aaa4fb089993c1fcb0fe58c"
format_stamp: "Formatted at 2026-09-27 14:00:48 on dist-test-slave-gkks"
server_key: "4e410467c74319f50133c40d6f05f8c8"
server_key_iv: "6ee4e74ce7d24049c45863d3704164b0"
server_key_version: "encryptionkey@0"
I20260927 14:00:48.700008 9568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:48.717190 9568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:48.721330 9568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:48.721616 9568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:48.722533 9568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:48.723907 9568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:48.724010 9568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:48.724164 9568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:48.724241 9568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:48.758355 9568 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:38575
I20260927 14:00:48.758414 9695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:38575 every 8 connection(s)
I20260927 14:00:48.759745 9568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 14:00:48.763569 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 9568
I20260927 14:00:48.763839 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 14:00:48.764412 22635 external_mini_cluster.cc:1428] Setting key 646b2e4ded6933df2b19ee27452fd2e2
Sep 27 14:00:48 dist-test-slave-gkks krb5kdc[9462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517648, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.193@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:00:48.780824 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:48.762674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:57443 (local address 127.22.26.254:37861)
0927 14:00:48.763439 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:48.763447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:48.764383 (+ 936us) server_negotiation.cc:407] Connection header received
0927 14:00:48.766286 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:48.766292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:48.766378 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:48.766503 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:48.768107 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:48.768845 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:48.770432 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:48.770840 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:48.770977 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:48.774996 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:48.775001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:48.775005 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:48.775038 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:48.777678 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:48.778419 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:48.778428 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:48.778431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:48.778546 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:48.778993 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:48.778998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:48.779001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:48.779348 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:48.779506 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:48.780231 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 14:00:48.780466 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":444,"threads_started":1}
I20260927 14:00:48.781809 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:48.763347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:37861 (local address 127.22.26.193:57443)
0927 14:00:48.764188 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:48.764236 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:48.766004 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:48.766760 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:48.766768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:48.767373 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:48.767938 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:48.767955 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:48.769013 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:48.769026 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:48.770282 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:48.770294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:48.771114 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:48.771122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:48.771360 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:48.772167 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:00:48.772189 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:48.774751 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:48.777830 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:48.777839 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:48.777857 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:48.778287 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:48.778658 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:48.778663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:48.778666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:48.778873 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:48.779477 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:48.779485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:48.779843 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 14:00:48.780113 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 14:00:48.780648 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":361,"threads_started":1}
I20260927 14:00:48.783526 9696 heartbeater.cc:347] Connected to a master server at 127.22.26.254:37861
I20260927 14:00:48.783843 9696 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:48.784587 9696 heartbeater.cc:511] Master 127.22.26.254:37861 requested a full tablet report, sending...
I20260927 14:00:48.787231 9507 ts_manager.cc:194] Registered new tserver with Master: 2786b0334aaa4fb089993c1fcb0fe58c (127.22.26.193:38575)
I20260927 14:00:48.789896 9507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.193@KRBTEST.COM'} at 127.22.26.193:57443
WARNING: no policy specified for kudu/127.22.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
Entry for principal kudu/127.22.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab.
I20260927 14:00:48.826839 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:37861
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:49.172394 9702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:49.172711 9702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:49.172879 9702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:49.180119 9702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:49.180264 9702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:49.180433 9702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 14:00:49.189996 9702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.194.keytab
--principal=kudu/127.22.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:37861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:49.191315 9702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:49.192585 9702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:49.205829 9710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:49.205899 9708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:49.205912 9707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:49.211983 9702 server_base.cc:1034] running on GCE node
I20260927 14:00:49.212620 9702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:49.213569 9702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:49.214807 9702 hybrid_clock.cc:648] HybridClock initialized: now 1790517649214770 us; error 47 us; skew 500 ppm
I20260927 14:00:49.215186 9702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:49 dist-test-slave-gkks krb5kdc[9462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517649, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:49.218310 9702 init.cc:377] Logged in from keytab as kudu/127.22.26.194@KRBTEST.COM (short username kudu)
I20260927 14:00:49.219887 9702 webserver.cc:460] Webserver started at http://127.22.26.194:36215/ using document root <none> and password file <none>
I20260927 14:00:49.220608 9702 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:49.220757 9702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:49.221076 9702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:49.224043 9702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "3406950e91a94ee7856e7aae513c26bc"
format_stamp: "Formatted at 2026-09-27 14:00:49 on dist-test-slave-gkks"
server_key: "f78e58acd8142f43b60724904dcddaa0"
server_key_iv: "b5ac049aaf376aef93dfed701da1ad7a"
server_key_version: "encryptionkey@0"
I20260927 14:00:49.224815 9702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "3406950e91a94ee7856e7aae513c26bc"
format_stamp: "Formatted at 2026-09-27 14:00:49 on dist-test-slave-gkks"
server_key: "f78e58acd8142f43b60724904dcddaa0"
server_key_iv: "b5ac049aaf376aef93dfed701da1ad7a"
server_key_version: "encryptionkey@0"
I20260927 14:00:49.230068 9702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 14:00:49.233639 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:49.234978 9702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:00:49.235208 9702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "3406950e91a94ee7856e7aae513c26bc"
format_stamp: "Formatted at 2026-09-27 14:00:49 on dist-test-slave-gkks"
server_key: "f78e58acd8142f43b60724904dcddaa0"
server_key_iv: "b5ac049aaf376aef93dfed701da1ad7a"
server_key_version: "encryptionkey@0"
I20260927 14:00:49.235430 9702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:49.272449 9702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:49.276901 9702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:49.277199 9702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:49.278213 9702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:49.279624 9702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:49.279726 9702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:49.279877 9702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:49.279950 9702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:49.312170 9702 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:37755
I20260927 14:00:49.312214 9829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:37755 every 8 connection(s)
I20260927 14:00:49.313462 9702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 14:00:49.322926 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 9702
I20260927 14:00:49.323139 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 14:00:49.323673 22635 external_mini_cluster.cc:1428] Setting key dda47286f23e05699c2d0eba67e7f08a
Sep 27 14:00:49 dist-test-slave-gkks krb5kdc[9462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517649, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.194@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:00:49.333804 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.316009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:60773 (local address 127.22.26.254:37861)
0927 14:00:49.316506 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:49.316511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:49.317487 (+ 976us) server_negotiation.cc:407] Connection header received
0927 14:00:49.319389 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:49.319396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:49.319486 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:49.319608 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:49.321096 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.321830 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.324096 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.324526 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.324639 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:49.328161 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:49.328166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:49.328169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:49.328206 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:49.330801 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:49.331560 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:49.331566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:49.331569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:49.331695 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:49.332115 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:49.332120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:49.332123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:49.332387 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:49.332537 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:49.333217 (+ 680us) server_negotiation.cc:299] Negotiation successful
0927 14:00:49.333434 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":260,"threads_started":1}
I20260927 14:00:49.334920 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.316622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:37861 (local address 127.22.26.194:60773)
0927 14:00:49.317319 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:49.317358 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:49.319045 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:49.319861 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:49.319870 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:49.320372 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:49.320916 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:49.320934 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.322002 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.322006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:49.323900 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:49.323915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.324774 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.324782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:49.325000 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:49.325753 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:00:49.325780 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:49.327935 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:49.330938 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:49.330946 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:49.330962 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:49.331436 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:49.331808 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:49.331813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:49.331817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:49.331998 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:49.332522 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:49.332529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:49.332866 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 14:00:49.333114 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 14:00:49.333610 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":268,"threads_started":1}
I20260927 14:00:49.336436 9830 heartbeater.cc:347] Connected to a master server at 127.22.26.254:37861
I20260927 14:00:49.336741 9830 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:49.337446 9830 heartbeater.cc:511] Master 127.22.26.254:37861 requested a full tablet report, sending...
I20260927 14:00:49.338913 9507 ts_manager.cc:194] Registered new tserver with Master: 3406950e91a94ee7856e7aae513c26bc (127.22.26.194:37755)
I20260927 14:00:49.340214 9507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.194@KRBTEST.COM'} at 127.22.26.194:60773
WARNING: no policy specified for kudu/127.22.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
Entry for principal kudu/127.22.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab.
I20260927 14:00:49.380605 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:37861
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:49.725086 9836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:49.725381 9836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:49.725564 9836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:49.733711 9836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:49.733827 9836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:49.734055 9836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 14:00:49.744735 9836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.195.keytab
--principal=kudu/127.22.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:37861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:49.746160 9836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:49.747606 9836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:49.761562 9841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:49.761525 9842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:49.762467 9844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:49.762603 9836 server_base.cc:1034] running on GCE node
I20260927 14:00:49.763294 9836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:49.764393 9836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:49.765611 9836 hybrid_clock.cc:648] HybridClock initialized: now 1790517649765554 us; error 57 us; skew 500 ppm
I20260927 14:00:49.765928 9836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:49 dist-test-slave-gkks krb5kdc[9462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517649, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:49.768954 9836 init.cc:377] Logged in from keytab as kudu/127.22.26.195@KRBTEST.COM (short username kudu)
I20260927 14:00:49.770291 9836 webserver.cc:460] Webserver started at http://127.22.26.195:46777/ using document root <none> and password file <none>
I20260927 14:00:49.770913 9836 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:49.771005 9836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:49.771324 9836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:49.774062 9836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "12c377fa19c9470a94c7faefcd3f0390"
format_stamp: "Formatted at 2026-09-27 14:00:49 on dist-test-slave-gkks"
server_key: "12bac757e714df0ba1c4ed5ed1b0e360"
server_key_iv: "d35975c7a82af7bcd0e2af9f375ac0c7"
server_key_version: "encryptionkey@0"
I20260927 14:00:49.774727 9836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "12c377fa19c9470a94c7faefcd3f0390"
format_stamp: "Formatted at 2026-09-27 14:00:49 on dist-test-slave-gkks"
server_key: "12bac757e714df0ba1c4ed5ed1b0e360"
server_key_iv: "d35975c7a82af7bcd0e2af9f375ac0c7"
server_key_version: "encryptionkey@0"
I20260927 14:00:49.779647 9836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 14:00:49.782902 9851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:49.784397 9836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:00:49.784555 9836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "12c377fa19c9470a94c7faefcd3f0390"
format_stamp: "Formatted at 2026-09-27 14:00:49 on dist-test-slave-gkks"
server_key: "12bac757e714df0ba1c4ed5ed1b0e360"
server_key_iv: "d35975c7a82af7bcd0e2af9f375ac0c7"
server_key_version: "encryptionkey@0"
I20260927 14:00:49.784857 9836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:49.793490 9696 heartbeater.cc:503] Master 127.22.26.254:37861 was elected leader, sending a full tablet report...
I20260927 14:00:49.800268 9836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:49.804579 9836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:49.804850 9836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:49.805868 9836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:49.807202 9836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:49.807296 9836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:49.807440 9836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:49.807514 9836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:49.839851 9836 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:44883
I20260927 14:00:49.839947 9963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:44883 every 8 connection(s)
I20260927 14:00:49.841354 9836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 14:00:49.851636 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 9836
I20260927 14:00:49.851872 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 14:00:49.852326 22635 external_mini_cluster.cc:1428] Setting key 3890ed7dcd3ef5218beec774fb9ac94a
Sep 27 14:00:49 dist-test-slave-gkks krb5kdc[9462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517649, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.195@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:00:49.861876 9966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.844140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:53199 (local address 127.22.26.254:37861)
0927 14:00:49.844628 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:49.844632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:49.845504 (+ 872us) server_negotiation.cc:407] Connection header received
0927 14:00:49.847402 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:49.847408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:49.847507 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:49.847627 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:49.849118 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.849905 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.851675 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.852205 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.852320 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:49.856143 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:49.856148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:49.856151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:49.856184 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:49.858620 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:49.859469 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:49.859474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:49.859477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:49.859596 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:49.860040 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:49.860045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:49.860047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:49.860323 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:49.860493 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:49.861183 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 14:00:49.861419 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":265,"threads_started":1}
I20260927 14:00:49.862811 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.844635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:37861 (local address 127.22.26.195:53199)
0927 14:00:49.845343 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:49.845386 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:49.847067 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:49.847867 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:49.847875 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:49.848400 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:49.848944 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:49.848959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.850061 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.850065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:49.851528 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:49.851542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.852503 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.852518 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:49.852861 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:49.853588 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:00:49.853619 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:49.855936 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:49.858777 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:49.858786 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:49.858803 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:49.859336 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:49.859711 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:00:49.859717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:00:49.859720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:00:49.859911 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:00:49.860458 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:49.860466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:49.860815 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 14:00:49.861072 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 14:00:49.861600 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":259,"threads_started":1}
I20260927 14:00:49.864261 9964 heartbeater.cc:347] Connected to a master server at 127.22.26.254:37861
I20260927 14:00:49.864523 9964 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:49.865175 9964 heartbeater.cc:511] Master 127.22.26.254:37861 requested a full tablet report, sending...
I20260927 14:00:49.866673 9507 ts_manager.cc:194] Registered new tserver with Master: 12c377fa19c9470a94c7faefcd3f0390 (127.22.26.195:44883)
I20260927 14:00:49.867842 9507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.26.195@KRBTEST.COM'} at 127.22.26.195:53199
I20260927 14:00:49.868290 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:00:49.884828 9966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.872331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.22.26.254:37861)
0927 14:00:49.872532 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:49.872542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:49.872945 (+ 403us) server_negotiation.cc:407] Connection header received
0927 14:00:49.873072 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:49.873077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:49.873163 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:49.873272 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:49.874603 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.875362 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.876684 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.877091 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.877188 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:49.879706 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:49.879711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:49.879714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:49.879755 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:49.882295 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:49.882953 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:49.882957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:49.882959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:49.883072 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:49.883498 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:49.883501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:49.883503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:49.883839 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:49.884014 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:49.884410 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 14:00:49.884580 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 14:00:49.888808 9507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:00:49.895829 9507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:00:49.923038 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.909340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:44883 (local address 127.0.0.1:46778)
0927 14:00:49.910579 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:49.910602 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:49.910749 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:49.911997 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:49.912005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:49.912046 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:49.912671 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:49.912688 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.915317 (+ 2629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.915325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:49.918457 (+ 3132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:49.918479 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.921459 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.921468 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:49.921915 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:49.921998 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 14:00:49.922277 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 14:00:49.922519 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"mutex_wait_us":1,"thread_start_us":923,"threads_started":1}
I20260927 14:00:49.923767 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.908275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48732 (local address 127.22.26.194:37755)
0927 14:00:49.908832 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:49.908842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:49.910456 (+ 1614us) server_negotiation.cc:407] Connection header received
0927 14:00:49.910595 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:49.910606 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:49.910921 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:49.912138 (+ 1217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:49.912456 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.914448 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.918428 (+ 3980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.920574 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.920753 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:49.921030 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:49.923190 (+ 2160us) server_negotiation.cc:299] Negotiation successful
0927 14:00:49.923367 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":263,"threads_started":1}
I20260927 14:00:49.923797 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.908727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.22.26.195:44883)
0927 14:00:49.910079 (+ 1352us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:49.910090 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:49.910901 (+ 811us) server_negotiation.cc:407] Connection header received
0927 14:00:49.911033 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:49.911044 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:49.911354 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:49.911551 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:49.912960 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.915043 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.918723 (+ 3680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.920814 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.920983 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:49.921273 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:49.923329 (+ 2056us) server_negotiation.cc:299] Negotiation successful
0927 14:00:49.923505 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":1004,"threads_started":1}
I20260927 14:00:49.924716 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.908041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:37755 (local address 127.0.0.1:48732)
0927 14:00:49.909882 (+ 1841us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:49.909920 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:49.910099 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:49.911202 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:49.911212 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:49.911254 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:49.911836 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:49.911864 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.914696 (+ 2832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.914705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:49.918158 (+ 3453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:49.918182 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.922440 (+ 4258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.922447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:49.922631 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:49.922675 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 14:00:49.922757 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 14:00:49.922960 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1598,"mutex_wait_us":35,"thread_start_us":1071,"threads_started":1}
I20260927 14:00:49.927966 9899 tablet_service.cc:1467] Processing CreateTablet for tablet d022aa2e19e44c7e89c386cbada3e0b1 (DEFAULT_TABLE table=test-table [id=788d6a22855e4354bae4bc612038ea2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:49.928493 9765 tablet_service.cc:1467] Processing CreateTablet for tablet d022aa2e19e44c7e89c386cbada3e0b1 (DEFAULT_TABLE table=test-table [id=788d6a22855e4354bae4bc612038ea2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:49.929628 9899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d022aa2e19e44c7e89c386cbada3e0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:49.929788 9765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d022aa2e19e44c7e89c386cbada3e0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:49.933005 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.909819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:38575 (local address 127.0.0.1:46068)
0927 14:00:49.922572 (+ 12753us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:49.922603 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:49.922735 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:49.925559 (+ 2824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:49.925562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:49.925585 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:49.925930 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:49.925942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.927890 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.927895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:49.930098 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:49.930114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.931985 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.931992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:49.932406 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:49.932439 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 14:00:49.932531 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 14:00:49.932665 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12605,"thread_start_us":12419,"threads_started":1}
I20260927 14:00:49.933750 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.910316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46068 (local address 127.22.26.193:38575)
0927 14:00:49.924730 (+ 14414us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:49.924739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:49.924768 (+ 29us) server_negotiation.cc:407] Connection header received
0927 14:00:49.924865 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:49.924873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:49.925141 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:49.925343 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:49.926146 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.927735 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.930270 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.931765 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.931919 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:49.932147 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:49.933281 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0927 14:00:49.933445 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14227,"thread_start_us":14056,"threads_started":1}
I20260927 14:00:49.937876 9631 tablet_service.cc:1467] Processing CreateTablet for tablet d022aa2e19e44c7e89c386cbada3e0b1 (DEFAULT_TABLE table=test-table [id=788d6a22855e4354bae4bc612038ea2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:49.939591 9631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d022aa2e19e44c7e89c386cbada3e0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:49.942639 9983 tablet_bootstrap.cc:492] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390: Bootstrap starting.
I20260927 14:00:49.946767 9984 tablet_bootstrap.cc:492] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc: Bootstrap starting.
I20260927 14:00:49.947090 9983 tablet_bootstrap.cc:654] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:49.948503 9983 log.cc:824] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:49.950189 9984 tablet_bootstrap.cc:654] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:49.951442 9984 log.cc:824] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:49.952275 9983 tablet_bootstrap.cc:492] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390: No bootstrap required, opened a new log
I20260927 14:00:49.952543 9983 ts_tablet_manager.cc:1397] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260927 14:00:49.954383 9987 tablet_bootstrap.cc:492] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c: Bootstrap starting.
I20260927 14:00:49.954427 9984 tablet_bootstrap.cc:492] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc: No bootstrap required, opened a new log
I20260927 14:00:49.954851 9984 ts_tablet_manager.cc:1397] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.008s
I20260927 14:00:49.957343 9987 tablet_bootstrap.cc:654] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:49.958278 9987 log.cc:824] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:49.961071 9987 tablet_bootstrap.cc:492] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c: No bootstrap required, opened a new log
I20260927 14:00:49.961330 9987 ts_tablet_manager.cc:1397] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 14:00:49.963840 9983 raft_consensus.cc:348] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3406950e91a94ee7856e7aae513c26bc" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37755 } } peers { permanent_uuid: "12c377fa19c9470a94c7faefcd3f0390" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44883 } } peers { permanent_uuid: "2786b0334aaa4fb089993c1fcb0fe58c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38575 } }
I20260927 14:00:49.964283 9983 raft_consensus.cc:374] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:49.964390 9983 raft_consensus.cc:729] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c377fa19c9470a94c7faefcd3f0390, State: Initialized, Role: FOLLOWER
I20260927 14:00:49.965014 9983 consensus_queue.cc:260] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3406950e91a94ee7856e7aae513c26bc" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37755 } } peers { permanent_uuid: "12c377fa19c9470a94c7faefcd3f0390" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44883 } } peers { permanent_uuid: "2786b0334aaa4fb089993c1fcb0fe58c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38575 } }
I20260927 14:00:49.965329 9984 raft_consensus.cc:348] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3406950e91a94ee7856e7aae513c26bc" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37755 } } peers { permanent_uuid: "12c377fa19c9470a94c7faefcd3f0390" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44883 } } peers { permanent_uuid: "2786b0334aaa4fb089993c1fcb0fe58c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38575 } }
I20260927 14:00:49.965790 9984 raft_consensus.cc:374] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:49.965896 9984 raft_consensus.cc:729] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3406950e91a94ee7856e7aae513c26bc, State: Initialized, Role: FOLLOWER
I20260927 14:00:49.966398 9964 heartbeater.cc:503] Master 127.22.26.254:37861 was elected leader, sending a full tablet report...
I20260927 14:00:49.966588 9984 consensus_queue.cc:260] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3406950e91a94ee7856e7aae513c26bc" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37755 } } peers { permanent_uuid: "12c377fa19c9470a94c7faefcd3f0390" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44883 } } peers { permanent_uuid: "2786b0334aaa4fb089993c1fcb0fe58c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38575 } }
I20260927 14:00:49.966845 9983 ts_tablet_manager.cc:1428] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260927 14:00:49.967769 9830 heartbeater.cc:503] Master 127.22.26.254:37861 was elected leader, sending a full tablet report...
I20260927 14:00:49.968401 9984 ts_tablet_manager.cc:1428] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 14:00:49.970304 9987 raft_consensus.cc:348] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3406950e91a94ee7856e7aae513c26bc" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37755 } } peers { permanent_uuid: "12c377fa19c9470a94c7faefcd3f0390" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44883 } } peers { permanent_uuid: "2786b0334aaa4fb089993c1fcb0fe58c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38575 } }
I20260927 14:00:49.970731 9987 raft_consensus.cc:374] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:49.970829 9987 raft_consensus.cc:729] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2786b0334aaa4fb089993c1fcb0fe58c, State: Initialized, Role: FOLLOWER
I20260927 14:00:49.971434 9987 consensus_queue.cc:260] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3406950e91a94ee7856e7aae513c26bc" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37755 } } peers { permanent_uuid: "12c377fa19c9470a94c7faefcd3f0390" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44883 } } peers { permanent_uuid: "2786b0334aaa4fb089993c1fcb0fe58c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38575 } }
I20260927 14:00:49.973212 9987 ts_tablet_manager.cc:1428] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260927 14:00:49.993723 9990 raft_consensus.cc:482] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:00:49.994051 9990 raft_consensus.cc:504] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3406950e91a94ee7856e7aae513c26bc" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37755 } } peers { permanent_uuid: "12c377fa19c9470a94c7faefcd3f0390" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44883 } } peers { permanent_uuid: "2786b0334aaa4fb089993c1fcb0fe58c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38575 } }
I20260927 14:00:49.995818 9990 leader_election.cc:288] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12c377fa19c9470a94c7faefcd3f0390 (127.22.26.195:44883), 2786b0334aaa4fb089993c1fcb0fe58c (127.22.26.193:38575)
I20260927 14:00:50.003392 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.996282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:45791 (local address 127.22.26.195:44883)
0927 14:00:49.996504 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:49.996513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:49.997509 (+ 996us) server_negotiation.cc:407] Connection header received
0927 14:00:49.997673 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:49.997676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:49.997768 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:49.997865 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:49.998534 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.999362 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.001574 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:50.002381 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.002564 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:50.002643 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:50.002985 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 14:00:50.003141 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 14:00:50.003398 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.996278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:46893 (local address 127.22.26.193:38575)
0927 14:00:49.996489 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:49.996497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:49.997384 (+ 887us) server_negotiation.cc:407] Connection header received
0927 14:00:49.997545 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:49.997551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:49.997633 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:49.997726 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:49.998311 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.999136 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.001559 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:50.002412 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.002589 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:50.002667 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:50.002972 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 14:00:50.003136 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 14:00:50.003543 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.996442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:44883 (local address 127.22.26.194:45791)
0927 14:00:49.997373 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:49.997397 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:49.997544 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:49.997906 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:49.997909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:49.997981 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:49.998372 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:49.998385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.999531 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.999535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:50.001415 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:50.001427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:50.002610 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.002620 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:50.002823 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:50.002851 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 14:00:50.002957 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 14:00:50.003086 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":30,"spinlock_wait_cycles":23424,"thread_start_us":513,"threads_started":1}
I20260927 14:00:50.003975 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:49.996412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:38575 (local address 127.22.26.194:46893)
0927 14:00:49.997272 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:49.997288 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:49.997406 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:49.997756 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:49.997760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:49.997784 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:49.998159 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:49.998169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:49.999291 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:49.999298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:50.001414 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:50.001426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:50.002635 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.002641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:50.002810 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:50.002836 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 14:00:50.002936 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 14:00:50.003068 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":5,"thread_start_us":449,"threads_started":1}
I20260927 14:00:50.004709 9919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022aa2e19e44c7e89c386cbada3e0b1" candidate_uuid: "3406950e91a94ee7856e7aae513c26bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c377fa19c9470a94c7faefcd3f0390" is_pre_election: true
I20260927 14:00:50.005173 9919 raft_consensus.cc:2454] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3406950e91a94ee7856e7aae513c26bc in term 0.
I20260927 14:00:50.005820 9719 leader_election.cc:302] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c377fa19c9470a94c7faefcd3f0390, 3406950e91a94ee7856e7aae513c26bc; no voters:
I20260927 14:00:50.005848 9651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022aa2e19e44c7e89c386cbada3e0b1" candidate_uuid: "3406950e91a94ee7856e7aae513c26bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2786b0334aaa4fb089993c1fcb0fe58c" is_pre_election: true
I20260927 14:00:50.006186 9990 raft_consensus.cc:2781] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:00:50.006318 9990 raft_consensus.cc:482] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:00:50.006425 9990 raft_consensus.cc:3037] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:50.006436 9651 raft_consensus.cc:2454] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3406950e91a94ee7856e7aae513c26bc in term 0.
I20260927 14:00:50.008846 9990 raft_consensus.cc:504] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3406950e91a94ee7856e7aae513c26bc" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37755 } } peers { permanent_uuid: "12c377fa19c9470a94c7faefcd3f0390" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44883 } } peers { permanent_uuid: "2786b0334aaa4fb089993c1fcb0fe58c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38575 } }
I20260927 14:00:50.009488 9990 leader_election.cc:288] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [CANDIDATE]: Term 1 election: Requested vote from peers 12c377fa19c9470a94c7faefcd3f0390 (127.22.26.195:44883), 2786b0334aaa4fb089993c1fcb0fe58c (127.22.26.193:38575)
I20260927 14:00:50.009884 9919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022aa2e19e44c7e89c386cbada3e0b1" candidate_uuid: "3406950e91a94ee7856e7aae513c26bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c377fa19c9470a94c7faefcd3f0390"
I20260927 14:00:50.010087 9919 raft_consensus.cc:3037] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:50.010028 9651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022aa2e19e44c7e89c386cbada3e0b1" candidate_uuid: "3406950e91a94ee7856e7aae513c26bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2786b0334aaa4fb089993c1fcb0fe58c"
I20260927 14:00:50.010262 9651 raft_consensus.cc:3037] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c [term 0 FOLLOWER]: Advancing to term 1
W20260927 14:00:50.012408 9697 tablet.cc:2369] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:00:50.012472 9919 raft_consensus.cc:2454] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3406950e91a94ee7856e7aae513c26bc in term 1.
I20260927 14:00:50.012564 9651 raft_consensus.cc:2454] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3406950e91a94ee7856e7aae513c26bc in term 1.
I20260927 14:00:50.012850 9719 leader_election.cc:302] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c377fa19c9470a94c7faefcd3f0390, 3406950e91a94ee7856e7aae513c26bc; no voters:
I20260927 14:00:50.013240 9990 raft_consensus.cc:2781] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:50.013649 9990 raft_consensus.cc:686] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [term 1 LEADER]: Becoming Leader. State: Replica: 3406950e91a94ee7856e7aae513c26bc, State: Running, Role: LEADER
I20260927 14:00:50.014094 9990 consensus_queue.cc:237] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3406950e91a94ee7856e7aae513c26bc" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37755 } } peers { permanent_uuid: "12c377fa19c9470a94c7faefcd3f0390" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44883 } } peers { permanent_uuid: "2786b0334aaa4fb089993c1fcb0fe58c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38575 } }
I20260927 14:00:50.018577 9505 catalog_manager.cc:5495] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3406950e91a94ee7856e7aae513c26bc (127.22.26.194). New cstate: current_term: 1 leader_uuid: "3406950e91a94ee7856e7aae513c26bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3406950e91a94ee7856e7aae513c26bc" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12c377fa19c9470a94c7faefcd3f0390" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2786b0334aaa4fb089993c1fcb0fe58c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38575 } health_report { overall_health: UNKNOWN } } }
I20260927 14:00:50.056653 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:50.051543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.22.26.194:37755)
0927 14:00:50.051737 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:50.051741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:50.051759 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:00:50.051960 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:50.051963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:50.052042 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:50.052179 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:00:50.052502 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:50.053286 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.054593 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:50.054993 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.055153 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:50.055659 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:00:50.056165 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:00:50.056293 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:50.056358 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 14:00:50.056421 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 14:00:50.065092 9651 raft_consensus.cc:1261] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c [term 1 FOLLOWER]: Refusing update from remote peer 3406950e91a94ee7856e7aae513c26bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:00:50.065092 9919 raft_consensus.cc:1261] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390 [term 1 FOLLOWER]: Refusing update from remote peer 3406950e91a94ee7856e7aae513c26bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 14:00:50.065523 9831 tablet.cc:2369] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:00:50.066175 9994 consensus_queue.cc:1035] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c377fa19c9470a94c7faefcd3f0390" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 44883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:00:50.066591 9990 consensus_queue.cc:1035] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2786b0334aaa4fb089993c1fcb0fe58c" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:00:50.095217 9506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:00:50.095584 9506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:00:50.099706 9506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbf4d59369384bb89ae40a2b7f452bab: Sending DeleteTablet for 3 replicas of tablet d022aa2e19e44c7e89c386cbada3e0b1
I20260927 14:00:50.100811 9765 tablet_service.cc:1514] Processing DeleteTablet for tablet d022aa2e19e44c7e89c386cbada3e0b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:50 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:48732
I20260927 14:00:50.101032 9899 tablet_service.cc:1514] Processing DeleteTablet for tablet d022aa2e19e44c7e89c386cbada3e0b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:50 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:46778
I20260927 14:00:50.101872 10007 tablet_replica.cc:330] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc: stopping tablet replica
I20260927 14:00:50.101805 9631 tablet_service.cc:1514] Processing DeleteTablet for tablet d022aa2e19e44c7e89c386cbada3e0b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:50 UTC) from {username='kudu', principal='kudu/127.22.26.254@KRBTEST.COM'} at 127.0.0.1:46068
I20260927 14:00:50.101936 10008 tablet_replica.cc:330] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390: stopping tablet replica
I20260927 14:00:50.102344 10007 raft_consensus.cc:2229] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:00:50.102327 10008 raft_consensus.cc:2229] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:00:50.102806 10008 raft_consensus.cc:2258] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:50.103955 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 9568
I20260927 14:00:50.103044 10007 raft_consensus.cc:2258] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:50.105031 10009 tablet_replica.cc:330] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c: stopping tablet replica
I20260927 14:00:50.105446 10009 raft_consensus.cc:2229] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:00:50.105554 10008 ts_tablet_manager.cc:1905] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:50.105894 10009 raft_consensus.cc:2258] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:50.107771 10007 ts_tablet_manager.cc:1905] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:50.108475 10009 ts_tablet_manager.cc:1905] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:50.112797 10008 ts_tablet_manager.cc:1918] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:50.112955 10008 log.cc:1196] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/d022aa2e19e44c7e89c386cbada3e0b1
I20260927 14:00:50.113523 10008 ts_tablet_manager.cc:1939] T d022aa2e19e44c7e89c386cbada3e0b1 P 12c377fa19c9470a94c7faefcd3f0390: Deleting consensus metadata
I20260927 14:00:50.115816 9492 catalog_manager.cc:4841] TS 12c377fa19c9470a94c7faefcd3f0390 (127.22.26.195:44883): tablet d022aa2e19e44c7e89c386cbada3e0b1 (table test-table [id=788d6a22855e4354bae4bc612038ea2e]) successfully deleted
I20260927 14:00:50.115891 10009 ts_tablet_manager.cc:1918] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:50.116043 10009 log.cc:1196] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-0/wal/wals/d022aa2e19e44c7e89c386cbada3e0b1
I20260927 14:00:50.116575 10009 ts_tablet_manager.cc:1939] T d022aa2e19e44c7e89c386cbada3e0b1 P 2786b0334aaa4fb089993c1fcb0fe58c: Deleting consensus metadata
Sep 27 14:00:50 dist-test-slave-gkks krb5kdc[9462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.193@KRBTEST.COM
I20260927 14:00:50.118301 9494 catalog_manager.cc:4841] TS 2786b0334aaa4fb089993c1fcb0fe58c (127.22.26.193:38575): tablet d022aa2e19e44c7e89c386cbada3e0b1 (table test-table [id=788d6a22855e4354bae4bc612038ea2e]) successfully deleted
I20260927 14:00:50.115041 10007 ts_tablet_manager.cc:1918] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:50.122210 10007 log.cc:1196] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/d022aa2e19e44c7e89c386cbada3e0b1
I20260927 14:00:50.122804 10007 ts_tablet_manager.cc:1939] T d022aa2e19e44c7e89c386cbada3e0b1 P 3406950e91a94ee7856e7aae513c26bc: Deleting consensus metadata
I20260927 14:00:50.124557 9491 catalog_manager.cc:4841] TS 3406950e91a94ee7856e7aae513c26bc (127.22.26.194:37755): tablet d022aa2e19e44c7e89c386cbada3e0b1 (table test-table [id=788d6a22855e4354bae4bc612038ea2e]) successfully deleted
I20260927 14:00:50.124773 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:50.105733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.22.26.193:38575)
0927 14:00:50.105918 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:50.105926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:50.106593 (+ 667us) server_negotiation.cc:407] Connection header received
0927 14:00:50.106796 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:50.106801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:50.106882 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:50.106973 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:50.109003 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:50.110137 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.112024 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:50.112664 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.112779 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:50.118544 (+ 5765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:50.118555 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:50.118567 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:50.118602 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:50.121380 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:50.122036 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:50.122045 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:50.122048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:50.122229 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:50.122738 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:50.122744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:50.122747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:50.123229 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:50.123511 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:50.124136 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 14:00:50.124357 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 14:00:50.125918 9691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:50.217062 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 9568
I20260927 14:00:50.235275 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 9702
Sep 27 14:00:50 dist-test-slave-gkks krb5kdc[9462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.194@KRBTEST.COM
I20260927 14:00:50.253968 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:50.236138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48756 (local address 127.22.26.194:37755)
0927 14:00:50.236356 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:50.236361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:50.236425 (+ 64us) server_negotiation.cc:407] Connection header received
0927 14:00:50.236618 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:50.236622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:50.236721 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:50.236812 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:50.238523 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:50.239668 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.241592 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:50.242027 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.242135 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:50.247679 (+ 5544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:50.247693 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:50.247708 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:50.247744 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:50.250984 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:50.251781 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:50.251796 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:50.251798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:50.251940 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:50.252329 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:50.252336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:50.252339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:50.252713 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:50.253074 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:50.253356 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 14:00:50.253613 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 14:00:50.254655 9825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:50.366655 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 9702
I20260927 14:00:50.385708 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 9836
Sep 27 14:00:50 dist-test-slave-gkks krb5kdc[9462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.195@KRBTEST.COM
I20260927 14:00:50.402570 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:50.386530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.22.26.195:44883)
0927 14:00:50.386712 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:50.386719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:50.386758 (+ 39us) server_negotiation.cc:407] Connection header received
0927 14:00:50.386972 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:50.386978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:50.387053 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:50.387211 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:50.388562 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:50.389730 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.391732 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:50.392264 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:50.392405 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:50.396852 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:50.396867 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:50.396883 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:50.396917 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:50.399694 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:50.400387 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:50.400396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:50.400400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:50.400540 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:50.400959 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:50.400966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:50.400969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:50.401324 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:50.401649 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:50.402014 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 14:00:50.402248 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 14:00:50.403276 9959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:50.493577 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 9836
I20260927 14:00:50.510753 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 9475
I20260927 14:00:50.511952 9537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:50.626314 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 9475
2026-09-27T14:00:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4306 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T14:00:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:00:50Z Disabled control of system clock
I20260927 14:00:50.703778 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:36029
--webserver_interface=127.22.26.254
--webserver_port=0
--builtin_ntp_servers=127.22.26.212:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:36029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 14:00:51.033870 10018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:51.034173 10018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:51.034253 10018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:51.041385 10018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:00:51.041494 10018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:51.041539 10018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:51.041589 10018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:00:51.041638 10018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:00:51.051666 10018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:36029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:36029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:51.052963 10018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:51.054772 10018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:51.071364 10026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:51.071460 10023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:51.071489 10024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:51.073249 10018 server_base.cc:1034] running on GCE node
I20260927 14:00:51.074038 10018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:51.075603 10018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:51.076814 10018 hybrid_clock.cc:648] HybridClock initialized: now 1790517651076751 us; error 55 us; skew 500 ppm
I20260927 14:00:51.077188 10018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 14:00:51.079582 10018 webserver.cc:460] Webserver started at http://127.22.26.254:42467/ using document root <none> and password file <none>
I20260927 14:00:51.080332 10018 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:51.080446 10018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:51.080785 10018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:51.083818 10018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "d8e69c5ee6334f8cbc82e9656e25192c"
format_stamp: "Formatted at 2026-09-27 14:00:51 on dist-test-slave-gkks"
server_key: "646d2e95acfbac892eca8c6627018db9"
server_key_iv: "243efb92657baeacd203c173d3b04326"
server_key_version: "encryptionkey@0"
I20260927 14:00:51.084646 10018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "d8e69c5ee6334f8cbc82e9656e25192c"
format_stamp: "Formatted at 2026-09-27 14:00:51 on dist-test-slave-gkks"
server_key: "646d2e95acfbac892eca8c6627018db9"
server_key_iv: "243efb92657baeacd203c173d3b04326"
server_key_version: "encryptionkey@0"
I20260927 14:00:51.089963 10018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 14:00:51.093885 10032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:51.095446 10018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 14:00:51.095616 10018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "d8e69c5ee6334f8cbc82e9656e25192c"
format_stamp: "Formatted at 2026-09-27 14:00:51 on dist-test-slave-gkks"
server_key: "646d2e95acfbac892eca8c6627018db9"
server_key_iv: "243efb92657baeacd203c173d3b04326"
server_key_version: "encryptionkey@0"
I20260927 14:00:51.095844 10018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:51.110126 10018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:51.110924 10018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:51.111265 10018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:51.130137 10018 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:36029
I20260927 14:00:51.130180 10083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:36029 every 8 connection(s)
I20260927 14:00:51.131508 10018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:00:51.135165 10084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:51.135600 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 10018
I20260927 14:00:51.135778 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:00:51.136240 22635 external_mini_cluster.cc:1428] Setting key 4e4704bf86d186a304e0a64c0d2ba793
I20260927 14:00:51.147502 10084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c: Bootstrap starting.
I20260927 14:00:51.149811 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:51.138117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.22.26.254:36029)
0927 14:00:51.139132 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:51.139148 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:51.139198 (+ 50us) server_negotiation.cc:407] Connection header received
0927 14:00:51.141182 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:51.141210 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:51.141658 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:51.142040 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:51.142629 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:51.143971 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:51.145370 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:51.146076 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:51.146291 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:51.147526 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:51.147534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:51.147549 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:51.148129 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:51.148219 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:51.148240 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:51.148374 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:51.148562 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 14:00:51.148867 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":346,"threads_started":1}
I20260927 14:00:51.151212 10084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:51.152254 10084 log.cc:824] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:51.155337 10084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c: No bootstrap required, opened a new log
I20260927 14:00:51.162850 10084 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e69c5ee6334f8cbc82e9656e25192c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36029 } }
I20260927 14:00:51.163317 10084 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:51.163424 10084 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e69c5ee6334f8cbc82e9656e25192c, State: Initialized, Role: FOLLOWER
I20260927 14:00:51.164036 10084 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e69c5ee6334f8cbc82e9656e25192c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36029 } }
I20260927 14:00:51.164287 10084 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:00:51.164433 10084 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:00:51.164619 10084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:51.167269 10084 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e69c5ee6334f8cbc82e9656e25192c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36029 } }
I20260927 14:00:51.167936 10084 leader_election.cc:302] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8e69c5ee6334f8cbc82e9656e25192c; no voters:
I20260927 14:00:51.168646 10084 leader_election.cc:288] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:00:51.168773 10089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:51.169456 10089 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [term 1 LEADER]: Becoming Leader. State: Replica: d8e69c5ee6334f8cbc82e9656e25192c, State: Running, Role: LEADER
I20260927 14:00:51.170083 10089 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e69c5ee6334f8cbc82e9656e25192c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36029 } }
I20260927 14:00:51.170955 10084 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:00:51.173051 10091 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8e69c5ee6334f8cbc82e9656e25192c. Latest consensus state: current_term: 1 leader_uuid: "d8e69c5ee6334f8cbc82e9656e25192c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e69c5ee6334f8cbc82e9656e25192c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36029 } } }
I20260927 14:00:51.173190 10090 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8e69c5ee6334f8cbc82e9656e25192c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e69c5ee6334f8cbc82e9656e25192c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 36029 } } }
I20260927 14:00:51.173460 10091 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:51.173583 10090 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:51.175575 10096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:00:51.180956 10096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:00:51.190091 10096 catalog_manager.cc:1347] Generated new cluster ID: 4f559335970e40479a037e7027b8a25e
I20260927 14:00:51.190225 10096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:00:51.209658 10096 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:00:51.210968 10096 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:00:51.223302 10096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c: Generated new TSK 0
I20260927 14:00:51.224188 10096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:00:51.233142 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.193:0
--local_ip_for_outbound_sockets=127.22.26.193
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36029
--builtin_ntp_servers=127.22.26.212:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 14:00:51.593976 10108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:51.594252 10108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:51.594338 10108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:51.601569 10108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:51.601662 10108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:51.601768 10108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.193
I20260927 14:00:51.610813 10108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.26.193
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.193
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:51.612068 10108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:51.613762 10108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:51.627115 10116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:51.627264 10113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:51.627233 10114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:51.628374 10108 server_base.cc:1034] running on GCE node
I20260927 14:00:51.629029 10108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:51.629956 10108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:51.631209 10108 hybrid_clock.cc:648] HybridClock initialized: now 1790517651631152 us; error 58 us; skew 500 ppm
I20260927 14:00:51.631572 10108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 14:00:51.633733 10108 webserver.cc:460] Webserver started at http://127.22.26.193:45089/ using document root <none> and password file <none>
I20260927 14:00:51.634418 10108 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:51.634552 10108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:51.634863 10108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:51.637722 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/data/instance:
uuid: "a0bda8a0500643d8a807e9d10b32d5ea"
format_stamp: "Formatted at 2026-09-27 14:00:51 on dist-test-slave-gkks"
server_key: "7bf4846fb93f5e71891f0c263fea87e7"
server_key_iv: "59cbcfcd084a2df0e35e575e518d7ce6"
server_key_version: "encryptionkey@0"
I20260927 14:00:51.638487 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance:
uuid: "a0bda8a0500643d8a807e9d10b32d5ea"
format_stamp: "Formatted at 2026-09-27 14:00:51 on dist-test-slave-gkks"
server_key: "7bf4846fb93f5e71891f0c263fea87e7"
server_key_iv: "59cbcfcd084a2df0e35e575e518d7ce6"
server_key_version: "encryptionkey@0"
I20260927 14:00:51.643564 10108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 14:00:51.647153 10122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:51.648463 10108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:00:51.648641 10108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/wal
uuid: "a0bda8a0500643d8a807e9d10b32d5ea"
format_stamp: "Formatted at 2026-09-27 14:00:51 on dist-test-slave-gkks"
server_key: "7bf4846fb93f5e71891f0c263fea87e7"
server_key_iv: "59cbcfcd084a2df0e35e575e518d7ce6"
server_key_version: "encryptionkey@0"
I20260927 14:00:51.648865 10108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:51.672096 10108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:51.672876 10108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:51.673180 10108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:51.674125 10108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:51.675776 10108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:51.675894 10108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:51.676017 10108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:51.676103 10108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:51.710734 10108 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.193:40701
I20260927 14:00:51.710824 10234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.193:40701 every 8 connection(s)
I20260927 14:00:51.712126 10108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/data/info.pb
I20260927 14:00:51.713466 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 10108
I20260927 14:00:51.713665 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-0/wal/instance
I20260927 14:00:51.714206 22635 external_mini_cluster.cc:1428] Setting key 51deae459315745ba335260c15c0adcd
I20260927 14:00:51.717778 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.194:0
--local_ip_for_outbound_sockets=127.22.26.194
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36029
--builtin_ntp_servers=127.22.26.212:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 14:00:51.725782 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:51.715033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.193:53857 (local address 127.22.26.254:36029)
0927 14:00:51.715574 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:51.715596 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:51.717089 (+ 1493us) server_negotiation.cc:407] Connection header received
0927 14:00:51.719037 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:51.719043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:51.719158 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:51.719286 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:51.720383 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:51.721125 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:51.722683 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:51.723293 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:51.723431 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:51.724819 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:51.724826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:51.724838 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:51.724858 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:51.724899 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:51.724912 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:51.725008 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:51.725229 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 14:00:51.725394 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":253,"threads_started":1}
I20260927 14:00:51.727017 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:51.715788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:36029 (local address 127.22.26.193:53857)
0927 14:00:51.716917 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:51.716963 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:51.718762 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:51.719534 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:51.719543 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:51.719640 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:51.720207 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:51.720224 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:51.721292 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:51.721297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:51.722532 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:51.722544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:51.723528 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:51.723535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:51.723780 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:51.724492 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:00:51.724495 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:51.724535 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:00:51.724616 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:51.725009 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:51.725023 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:51.725033 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 14:00:51.725217 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 14:00:51.725645 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":365,"threads_started":1}
I20260927 14:00:51.728654 10235 heartbeater.cc:347] Connected to a master server at 127.22.26.254:36029
I20260927 14:00:51.728976 10235 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:51.729861 10235 heartbeater.cc:511] Master 127.22.26.254:36029 requested a full tablet report, sending...
I20260927 14:00:51.732164 10049 ts_manager.cc:194] Registered new tserver with Master: a0bda8a0500643d8a807e9d10b32d5ea (127.22.26.193:40701)
I20260927 14:00:51.734295 10049 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.26.193:53857
W20260927 14:00:52.055177 10239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:52.055449 10239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:52.055533 10239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:52.062952 10239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:52.063055 10239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:52.063220 10239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.194
I20260927 14:00:52.074451 10239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.26.194
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.194
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:52.076267 10239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:52.078295 10239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:52.091378 10244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:52.091495 10245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:52.091790 10247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:52.093011 10239 server_base.cc:1034] running on GCE node
I20260927 14:00:52.093792 10239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:52.094868 10239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:52.096129 10239 hybrid_clock.cc:648] HybridClock initialized: now 1790517652096082 us; error 61 us; skew 500 ppm
I20260927 14:00:52.096485 10239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 14:00:52.098999 10239 webserver.cc:460] Webserver started at http://127.22.26.194:35721/ using document root <none> and password file <none>
I20260927 14:00:52.099901 10239 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:52.100045 10239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:52.100395 10239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:52.103641 10239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/data/instance:
uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831"
format_stamp: "Formatted at 2026-09-27 14:00:52 on dist-test-slave-gkks"
server_key: "c51326f53d0f601646d10678ed6ae43e"
server_key_iv: "8c268b5b25bc6d337e827065270a3772"
server_key_version: "encryptionkey@0"
I20260927 14:00:52.104501 10239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance:
uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831"
format_stamp: "Formatted at 2026-09-27 14:00:52 on dist-test-slave-gkks"
server_key: "c51326f53d0f601646d10678ed6ae43e"
server_key_iv: "8c268b5b25bc6d337e827065270a3772"
server_key_version: "encryptionkey@0"
I20260927 14:00:52.110260 10239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 14:00:52.114182 10253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:52.115620 10239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:00:52.115813 10239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/wal
uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831"
format_stamp: "Formatted at 2026-09-27 14:00:52 on dist-test-slave-gkks"
server_key: "c51326f53d0f601646d10678ed6ae43e"
server_key_iv: "8c268b5b25bc6d337e827065270a3772"
server_key_version: "encryptionkey@0"
I20260927 14:00:52.116057 10239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:52.142335 10239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:52.143329 10239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:52.143666 10239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:52.144814 10239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:52.146683 10239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:52.146812 10239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:52.146934 10239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:52.147012 10239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:52.182933 10239 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.194:35615
I20260927 14:00:52.183060 10365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.194:35615 every 8 connection(s)
I20260927 14:00:52.184350 10239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/data/info.pb
I20260927 14:00:52.186836 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 10239
I20260927 14:00:52.187027 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/wal/instance
I20260927 14:00:52.187600 22635 external_mini_cluster.cc:1428] Setting key ef390cdf17254a3c6cfb2c52c740ce14
I20260927 14:00:52.190965 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.26.195:0
--local_ip_for_outbound_sockets=127.22.26.195
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36029
--builtin_ntp_servers=127.22.26.212:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 14:00:52.199545 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:52.186835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.194:37311 (local address 127.22.26.254:36029)
0927 14:00:52.187223 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:52.187232 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:52.189010 (+ 1778us) server_negotiation.cc:407] Connection header received
0927 14:00:52.191589 (+ 2579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:52.191595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:52.191665 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:52.191762 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:52.193218 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.193945 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.196452 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.196892 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.197016 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:52.198677 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:52.198683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:52.198686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:52.198761 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:52.198805 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:52.198820 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:52.198897 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:52.199203 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 14:00:52.199292 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 14:00:52.200929 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:52.187542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:36029 (local address 127.22.26.194:37311)
0927 14:00:52.188840 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:52.188901 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:52.191282 (+ 2381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:52.192038 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:52.192053 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:52.192178 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:52.192999 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:52.193025 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.194108 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.194116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:52.196293 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:52.196312 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.197123 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.197134 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:52.197486 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:52.198283 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:00:52.198288 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:52.198342 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:00:52.198450 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:52.198916 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:52.198929 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:52.198940 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 14:00:52.199189 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 14:00:52.199636 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":342,"threads_started":1}
I20260927 14:00:52.202474 10366 heartbeater.cc:347] Connected to a master server at 127.22.26.254:36029
I20260927 14:00:52.202791 10366 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:52.203588 10366 heartbeater.cc:511] Master 127.22.26.254:36029 requested a full tablet report, sending...
I20260927 14:00:52.205124 10049 ts_manager.cc:194] Registered new tserver with Master: fb138f0dff9c4b0f8c4d3bb7abca5831 (127.22.26.194:35615)
I20260927 14:00:52.206386 10049 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.26.194:37311
W20260927 14:00:52.519470 10369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:52.519727 10369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:52.519809 10369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:52.526700 10369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:52.526793 10369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:52.526891 10369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.26.195
I20260927 14:00:52.535437 10369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.26.195
--webserver_port=0
--tserver_master_addrs=127.22.26.254:36029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.26.195
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:52.536662 10369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:52.538368 10369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:52.551594 10374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:52.551539 10375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:52.551633 10377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:52.553093 10369 server_base.cc:1034] running on GCE node
I20260927 14:00:52.553612 10369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:52.554486 10369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:52.555687 10369 hybrid_clock.cc:648] HybridClock initialized: now 1790517652555613 us; error 49 us; skew 500 ppm
I20260927 14:00:52.555948 10369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 14:00:52.558126 10369 webserver.cc:460] Webserver started at http://127.22.26.195:42551/ using document root <none> and password file <none>
I20260927 14:00:52.558755 10369 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:52.558837 10369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:52.559118 10369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:52.561825 10369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/data/instance:
uuid: "cee81b89a78d4ec2851f1c46ad4972aa"
format_stamp: "Formatted at 2026-09-27 14:00:52 on dist-test-slave-gkks"
server_key: "3a86954fb87c33ef3cf6168db03daabf"
server_key_iv: "c5751283b44e2467d584cc22a8f221c5"
server_key_version: "encryptionkey@0"
I20260927 14:00:52.562520 10369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance:
uuid: "cee81b89a78d4ec2851f1c46ad4972aa"
format_stamp: "Formatted at 2026-09-27 14:00:52 on dist-test-slave-gkks"
server_key: "3a86954fb87c33ef3cf6168db03daabf"
server_key_iv: "c5751283b44e2467d584cc22a8f221c5"
server_key_version: "encryptionkey@0"
I20260927 14:00:52.567427 10369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 14:00:52.571053 10383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:52.572508 10369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:00:52.572629 10369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/wal
uuid: "cee81b89a78d4ec2851f1c46ad4972aa"
format_stamp: "Formatted at 2026-09-27 14:00:52 on dist-test-slave-gkks"
server_key: "3a86954fb87c33ef3cf6168db03daabf"
server_key_iv: "c5751283b44e2467d584cc22a8f221c5"
server_key_version: "encryptionkey@0"
I20260927 14:00:52.572817 10369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:52.585692 10369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:52.586378 10369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:52.586617 10369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:52.587540 10369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:00:52.588887 10369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:00:52.588969 10369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:52.589048 10369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:00:52.589103 10369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:52.622296 10369 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.195:36073
I20260927 14:00:52.622385 10495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.195:36073 every 8 connection(s)
I20260927 14:00:52.623886 10369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/data/info.pb
I20260927 14:00:52.629812 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 10369
I20260927 14:00:52.630029 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/wal/instance
I20260927 14:00:52.630641 22635 external_mini_cluster.cc:1428] Setting key 10acbf65925619c516dc3ca79a178095
I20260927 14:00:52.636051 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:52.626279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:54321 (local address 127.22.26.254:36029)
0927 14:00:52.626471 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:52.626476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:52.627863 (+ 1387us) server_negotiation.cc:407] Connection header received
0927 14:00:52.629785 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:52.629791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:52.629884 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:52.630002 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:52.631398 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.632133 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.633642 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.634028 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.634120 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:52.635415 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:52.635420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:52.635423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:52.635438 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:52.635467 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:52.635479 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:52.635549 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:52.635754 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 14:00:52.635826 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 14:00:52.636956 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:52.626766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:36029 (local address 127.22.26.195:54321)
0927 14:00:52.627707 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:52.627751 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:52.629518 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:52.630274 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:52.630288 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:52.630407 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:00:52.631219 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:52.631238 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.632310 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.632314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:52.633517 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:52.633529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.634219 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.634224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:52.634410 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:52.635072 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:00:52.635075 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:00:52.635168 (+ 93us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:00:52.635252 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:00:52.635562 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:00:52.635568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:00:52.635573 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 14:00:52.635699 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 14:00:52.635957 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":308,"threads_started":1}
I20260927 14:00:52.638257 10496 heartbeater.cc:347] Connected to a master server at 127.22.26.254:36029
I20260927 14:00:52.638504 10496 heartbeater.cc:464] Registering TS with master...
I20260927 14:00:52.639189 10496 heartbeater.cc:511] Master 127.22.26.254:36029 requested a full tablet report, sending...
I20260927 14:00:52.640475 10049 ts_manager.cc:194] Registered new tserver with Master: cee81b89a78d4ec2851f1c46ad4972aa (127.22.26.195:36073)
I20260927 14:00:52.641207 10049 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.26.195:54321
I20260927 14:00:52.649495 22635 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:00:52.659442 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:52.653669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57320 (local address 127.22.26.254:36029)
0927 14:00:52.653930 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:52.653949 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:52.654312 (+ 363us) server_negotiation.cc:407] Connection header received
0927 14:00:52.654436 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:52.654440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:52.654518 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:52.654611 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:52.655067 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.655890 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.657177 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.657616 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.657720 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:52.658644 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:52.658649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:52.658652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:52.658669 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:52.658700 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:52.658712 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:52.658783 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:52.658978 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 14:00:52.659048 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 14:00:52.663267 10049 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:00:52.670320 10049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:00:52.698407 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:52.683601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.22.26.194:35615)
0927 14:00:52.684374 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:52.684383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:52.685120 (+ 737us) server_negotiation.cc:407] Connection header received
0927 14:00:52.685601 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:52.685613 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:52.685934 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:52.686489 (+ 555us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:52.687762 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.689842 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.693583 (+ 3741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.695607 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.695750 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:52.696137 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:52.697804 (+ 1667us) server_negotiation.cc:299] Negotiation successful
0927 14:00:52.697992 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":464,"threads_started":1}
I20260927 14:00:52.699378 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:52.683294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:35615 (local address 127.0.0.1:58022)
0927 14:00:52.684964 (+ 1670us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:52.685003 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:52.685291 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:52.686906 (+ 1615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:52.686917 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:52.686958 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:52.687508 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:52.687524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.690091 (+ 2567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.690099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:52.693297 (+ 3198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:52.693319 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.696371 (+ 3052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.696381 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:52.696684 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:52.696760 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 14:00:52.698760 (+ 2000us) client_negotiation.cc:240] Negotiation successful
0927 14:00:52.698976 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1390,"thread_start_us":13776,"threads_started":1}
I20260927 14:00:52.703823 10301 tablet_service.cc:1467] Processing CreateTablet for tablet 90406ba02c1e4f36b55475e417fe17e1 (DEFAULT_TABLE table=test-table [id=f3db14a9f8004800a8a40b1fed0c5135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:52.705319 10301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90406ba02c1e4f36b55475e417fe17e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:52.705799 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:52.685100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40701 (local address 127.0.0.1:42718)
0927 14:00:52.697693 (+ 12593us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:52.697712 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:52.697807 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:52.699215 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:52.699223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:52.699255 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:52.699887 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:52.699905 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.701555 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.701563 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:52.703552 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:52.703564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.705056 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.705063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:52.705219 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:52.705252 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 14:00:52.705343 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 14:00:52.705465 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12472,"mutex_wait_us":69,"thread_start_us":12775,"threads_started":1}
I20260927 14:00:52.705899 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:52.685712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.22.26.193:40701)
0927 14:00:52.697829 (+ 12117us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:52.697842 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:52.697883 (+ 41us) server_negotiation.cc:407] Connection header received
0927 14:00:52.698015 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:52.698027 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:52.698390 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:52.698856 (+ 466us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:52.700086 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.701388 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.703799 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.704851 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.704991 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:52.705266 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:52.705397 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 14:00:52.705563 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11943,"thread_start_us":11809,"threads_started":1}
I20260927 14:00:52.708182 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:52.683596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.195:36073 (local address 127.0.0.1:45978)
0927 14:00:52.697441 (+ 13845us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:52.697466 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:52.697604 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:52.700245 (+ 2641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:52.700250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:52.700274 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:52.700633 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:52.700645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.702553 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.702562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:52.704627 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:52.704643 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.707367 (+ 2724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.707374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:52.707545 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:52.707580 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 14:00:52.707713 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 14:00:52.707834 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13665,"mutex_wait_us":4,"thread_start_us":940,"threads_started":1}
I20260927 14:00:52.708297 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:52.684244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.22.26.195:36073)
0927 14:00:52.699596 (+ 15352us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:52.699604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:52.699641 (+ 37us) server_negotiation.cc:407] Connection header received
0927 14:00:52.699746 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:52.699754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:52.699984 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:52.700209 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:52.700815 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.702364 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.704827 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:52.706846 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:52.706978 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:52.707242 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:52.707758 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 14:00:52.707912 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15138,"thread_start_us":14973,"threads_started":1}
I20260927 14:00:52.710160 10170 tablet_service.cc:1467] Processing CreateTablet for tablet 90406ba02c1e4f36b55475e417fe17e1 (DEFAULT_TABLE table=test-table [id=f3db14a9f8004800a8a40b1fed0c5135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:52.711639 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90406ba02c1e4f36b55475e417fe17e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:52.711897 10431 tablet_service.cc:1467] Processing CreateTablet for tablet 90406ba02c1e4f36b55475e417fe17e1 (DEFAULT_TABLE table=test-table [id=f3db14a9f8004800a8a40b1fed0c5135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:00:52.713218 10431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90406ba02c1e4f36b55475e417fe17e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:52.725907 10514 tablet_bootstrap.cc:492] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831: Bootstrap starting.
I20260927 14:00:52.725934 10515 tablet_bootstrap.cc:492] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa: Bootstrap starting.
I20260927 14:00:52.730093 10515 tablet_bootstrap.cc:654] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:52.730088 10514 tablet_bootstrap.cc:654] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:52.731127 10515 log.cc:824] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:52.731519 10514 log.cc:824] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:52.732024 10516 tablet_bootstrap.cc:492] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea: Bootstrap starting.
I20260927 14:00:52.736132 10516 tablet_bootstrap.cc:654] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:52.737334 10516 log.cc:824] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:52.737556 10515 tablet_bootstrap.cc:492] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa: No bootstrap required, opened a new log
I20260927 14:00:52.737839 10515 ts_tablet_manager.cc:1397] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260927 14:00:52.737972 10235 heartbeater.cc:503] Master 127.22.26.254:36029 was elected leader, sending a full tablet report...
I20260927 14:00:52.738075 10514 tablet_bootstrap.cc:492] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831: No bootstrap required, opened a new log
I20260927 14:00:52.738369 10514 ts_tablet_manager.cc:1397] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260927 14:00:52.745338 10516 tablet_bootstrap.cc:492] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea: No bootstrap required, opened a new log
I20260927 14:00:52.745601 10516 ts_tablet_manager.cc:1397] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260927 14:00:52.746483 10515 raft_consensus.cc:348] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35615 } } peers { permanent_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36073 } } peers { permanent_uuid: "a0bda8a0500643d8a807e9d10b32d5ea" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40701 } }
I20260927 14:00:52.746843 10515 raft_consensus.cc:374] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:52.746938 10515 raft_consensus.cc:729] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee81b89a78d4ec2851f1c46ad4972aa, State: Initialized, Role: FOLLOWER
I20260927 14:00:52.747565 10515 consensus_queue.cc:260] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35615 } } peers { permanent_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36073 } } peers { permanent_uuid: "a0bda8a0500643d8a807e9d10b32d5ea" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40701 } }
I20260927 14:00:52.748631 10496 heartbeater.cc:503] Master 127.22.26.254:36029 was elected leader, sending a full tablet report...
I20260927 14:00:52.749321 10515 ts_tablet_manager.cc:1428] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260927 14:00:52.752071 10514 raft_consensus.cc:348] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35615 } } peers { permanent_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36073 } } peers { permanent_uuid: "a0bda8a0500643d8a807e9d10b32d5ea" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40701 } }
I20260927 14:00:52.752560 10514 raft_consensus.cc:374] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:52.752648 10514 raft_consensus.cc:729] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb138f0dff9c4b0f8c4d3bb7abca5831, State: Initialized, Role: FOLLOWER
I20260927 14:00:52.753212 10514 consensus_queue.cc:260] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35615 } } peers { permanent_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36073 } } peers { permanent_uuid: "a0bda8a0500643d8a807e9d10b32d5ea" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40701 } }
I20260927 14:00:52.754191 10366 heartbeater.cc:503] Master 127.22.26.254:36029 was elected leader, sending a full tablet report...
I20260927 14:00:52.754897 10514 ts_tablet_manager.cc:1428] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260927 14:00:52.755975 10516 raft_consensus.cc:348] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35615 } } peers { permanent_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36073 } } peers { permanent_uuid: "a0bda8a0500643d8a807e9d10b32d5ea" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40701 } }
I20260927 14:00:52.756418 10516 raft_consensus.cc:374] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:52.756510 10516 raft_consensus.cc:729] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0bda8a0500643d8a807e9d10b32d5ea, State: Initialized, Role: FOLLOWER
I20260927 14:00:52.757012 10516 consensus_queue.cc:260] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35615 } } peers { permanent_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36073 } } peers { permanent_uuid: "a0bda8a0500643d8a807e9d10b32d5ea" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40701 } }
I20260927 14:00:52.758544 10516 ts_tablet_manager.cc:1428] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260927 14:00:52.875649 10497 tablet.cc:2369] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:00:52.936482 10367 tablet.cc:2369] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:00:52.964641 10236 tablet.cc:2369] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:00:53.021869 10520 raft_consensus.cc:482] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:00:53.022230 10520 raft_consensus.cc:504] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35615 } } peers { permanent_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36073 } } peers { permanent_uuid: "a0bda8a0500643d8a807e9d10b32d5ea" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40701 } }
I20260927 14:00:53.024830 10520 leader_election.cc:288] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb138f0dff9c4b0f8c4d3bb7abca5831 (127.22.26.194:35615), a0bda8a0500643d8a807e9d10b32d5ea (127.22.26.193:40701)
I20260927 14:00:53.033420 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.025685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:58227 (local address 127.22.26.193:40701)
0927 14:00:53.025876 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:53.025884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:53.025905 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:00:53.026146 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:53.026152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:53.026248 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:53.026312 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:53.027117 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.028444 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.030969 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.032237 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.032398 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:53.032481 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:53.032976 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 14:00:53.033131 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:00:53.034330 10523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.025226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40701 (local address 127.22.26.195:58227)
0927 14:00:53.025828 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:53.025845 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:53.025968 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:53.026460 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:53.026463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:53.026482 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:53.026916 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:53.026926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.028664 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.028676 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:53.030769 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:53.030781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.032592 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.032598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:53.032760 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:53.032800 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 14:00:53.033685 (+ 885us) client_negotiation.cc:240] Negotiation successful
0927 14:00:53.033830 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":2,"thread_start_us":724,"threads_started":1}
I20260927 14:00:53.034737 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.025099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.194:35615 (local address 127.22.26.195:59693)
0927 14:00:53.025532 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:53.025555 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 14:00:53.025708 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:00:53.026632 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:00:53.026636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:00:53.026681 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:00:53.026915 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:53.026926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.029966 (+ 3040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.029972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:53.032014 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:00:53.032028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.033884 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.033892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:00:53.034106 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:53.034152 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 14:00:53.034252 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 14:00:53.034417 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":1}
I20260927 14:00:53.035599 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90406ba02c1e4f36b55475e417fe17e1" candidate_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0bda8a0500643d8a807e9d10b32d5ea" is_pre_election: true
I20260927 14:00:53.036233 10190 raft_consensus.cc:2454] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee81b89a78d4ec2851f1c46ad4972aa in term 0.
I20260927 14:00:53.036962 10385 leader_election.cc:302] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0bda8a0500643d8a807e9d10b32d5ea, cee81b89a78d4ec2851f1c46ad4972aa; no voters:
I20260927 14:00:53.037145 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.025960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.26.195:59693 (local address 127.22.26.194:35615)
0927 14:00:53.026175 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:53.026183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:53.026214 (+ 31us) server_negotiation.cc:407] Connection header received
0927 14:00:53.026374 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:53.026381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:53.026472 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:53.026577 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:00:53.028674 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.029780 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.032215 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.033501 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.033701 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:53.033789 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:53.036723 (+ 2934us) server_negotiation.cc:299] Negotiation successful
0927 14:00:53.036867 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 14:00:53.037386 10520 raft_consensus.cc:2781] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:00:53.037525 10520 raft_consensus.cc:482] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:00:53.037639 10520 raft_consensus.cc:3037] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:53.038077 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90406ba02c1e4f36b55475e417fe17e1" candidate_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831" is_pre_election: true
I20260927 14:00:53.038589 10321 raft_consensus.cc:2454] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee81b89a78d4ec2851f1c46ad4972aa in term 0.
I20260927 14:00:53.040441 10520 raft_consensus.cc:504] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35615 } } peers { permanent_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36073 } } peers { permanent_uuid: "a0bda8a0500643d8a807e9d10b32d5ea" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40701 } }
I20260927 14:00:53.041097 10520 leader_election.cc:288] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [CANDIDATE]: Term 1 election: Requested vote from peers fb138f0dff9c4b0f8c4d3bb7abca5831 (127.22.26.194:35615), a0bda8a0500643d8a807e9d10b32d5ea (127.22.26.193:40701)
I20260927 14:00:53.041462 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90406ba02c1e4f36b55475e417fe17e1" candidate_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831"
I20260927 14:00:53.041603 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90406ba02c1e4f36b55475e417fe17e1" candidate_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0bda8a0500643d8a807e9d10b32d5ea"
I20260927 14:00:53.041676 10321 raft_consensus.cc:3037] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:53.041838 10190 raft_consensus.cc:3037] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:53.043916 10321 raft_consensus.cc:2454] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee81b89a78d4ec2851f1c46ad4972aa in term 1.
I20260927 14:00:53.044224 10190 raft_consensus.cc:2454] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee81b89a78d4ec2851f1c46ad4972aa in term 1.
I20260927 14:00:53.044339 10386 leader_election.cc:302] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cee81b89a78d4ec2851f1c46ad4972aa, fb138f0dff9c4b0f8c4d3bb7abca5831; no voters:
I20260927 14:00:53.044627 10520 raft_consensus.cc:2781] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:53.045096 10520 raft_consensus.cc:686] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [term 1 LEADER]: Becoming Leader. State: Replica: cee81b89a78d4ec2851f1c46ad4972aa, State: Running, Role: LEADER
I20260927 14:00:53.045650 10520 consensus_queue.cc:237] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35615 } } peers { permanent_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36073 } } peers { permanent_uuid: "a0bda8a0500643d8a807e9d10b32d5ea" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40701 } }
I20260927 14:00:53.051642 10049 catalog_manager.cc:5495] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa reported cstate change: term changed from 0 to 1, leader changed from <none> to cee81b89a78d4ec2851f1c46ad4972aa (127.22.26.195). New cstate: current_term: 1 leader_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cee81b89a78d4ec2851f1c46ad4972aa" member_type: VOTER last_known_addr { host: "127.22.26.195" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0bda8a0500643d8a807e9d10b32d5ea" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40701 } health_report { overall_health: UNKNOWN } } }
I20260927 14:00:53.081388 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.076223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.22.26.195:36073)
0927 14:00:53.076383 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:53.076387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:53.076555 (+ 168us) server_negotiation.cc:407] Connection header received
0927 14:00:53.076787 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:53.076791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:53.076865 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:53.077030 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:00:53.077326 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.078093 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.079496 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.079906 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.079994 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:53.080423 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:00:53.080933 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:00:53.081036 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:53.081135 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 14:00:53.081194 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:00:53.089730 10321 raft_consensus.cc:1261] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831 [term 1 FOLLOWER]: Refusing update from remote peer cee81b89a78d4ec2851f1c46ad4972aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:00:53.089740 10190 raft_consensus.cc:1261] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea [term 1 FOLLOWER]: Refusing update from remote peer cee81b89a78d4ec2851f1c46ad4972aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:00:53.090837 10524 consensus_queue.cc:1035] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0bda8a0500643d8a807e9d10b32d5ea" member_type: VOTER last_known_addr { host: "127.22.26.193" port: 40701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:00:53.091254 10520 consensus_queue.cc:1035] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb138f0dff9c4b0f8c4d3bb7abca5831" member_type: VOTER last_known_addr { host: "127.22.26.194" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:00:53.116163 10048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:00:53.116473 10048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:00:53.120409 10048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8e69c5ee6334f8cbc82e9656e25192c: Sending DeleteTablet for 3 replicas of tablet 90406ba02c1e4f36b55475e417fe17e1
I20260927 14:00:53.121524 10301 tablet_service.cc:1514] Processing DeleteTablet for tablet 90406ba02c1e4f36b55475e417fe17e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:53 UTC) from {username='slave'} at 127.0.0.1:58022
I20260927 14:00:53.121583 10431 tablet_service.cc:1514] Processing DeleteTablet for tablet 90406ba02c1e4f36b55475e417fe17e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:53 UTC) from {username='slave'} at 127.0.0.1:45978
I20260927 14:00:53.122243 10170 tablet_service.cc:1514] Processing DeleteTablet for tablet 90406ba02c1e4f36b55475e417fe17e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:00:53 UTC) from {username='slave'} at 127.0.0.1:42718
I20260927 14:00:53.123328 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 10108
I20260927 14:00:53.143236 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.125045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.22.26.193:40701)
0927 14:00:53.125291 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:53.125301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:53.127622 (+ 2321us) server_negotiation.cc:407] Connection header received
0927 14:00:53.127826 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:53.127834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:53.127984 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:53.128123 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:53.128952 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.130182 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.132081 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.132745 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.132831 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:53.141547 (+ 8716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:53.141558 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:53.141576 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:53.142331 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:53.142399 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:53.142425 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:53.142510 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:53.142733 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 14:00:53.142849 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 14:00:53.144409 10230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:53.157282 10538 tablet_replica.cc:330] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831: stopping tablet replica
I20260927 14:00:53.157895 10537 tablet_replica.cc:330] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa: stopping tablet replica
I20260927 14:00:53.158499 10537 raft_consensus.cc:2229] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:00:53.159036 10538 raft_consensus.cc:2229] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:00:53.159646 10538 raft_consensus.cc:2258] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:53.159943 10537 raft_consensus.cc:2258] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:00:53.161715 10537 ts_tablet_manager.cc:1905] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:53.163542 10538 ts_tablet_manager.cc:1905] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:00:53.169373 10537 ts_tablet_manager.cc:1918] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:53.169535 10537 log.cc:1196] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-2/wal/wals/90406ba02c1e4f36b55475e417fe17e1
I20260927 14:00:53.170051 10537 ts_tablet_manager.cc:1939] T 90406ba02c1e4f36b55475e417fe17e1 P cee81b89a78d4ec2851f1c46ad4972aa: Deleting consensus metadata
I20260927 14:00:53.170606 10538 ts_tablet_manager.cc:1918] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:00:53.170831 10538 log.cc:1196] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831: Deleting WAL directory at /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790517330258165-22635-0/minicluster-data/ts-1/wal/wals/90406ba02c1e4f36b55475e417fe17e1
I20260927 14:00:53.171445 10538 ts_tablet_manager.cc:1939] T 90406ba02c1e4f36b55475e417fe17e1 P fb138f0dff9c4b0f8c4d3bb7abca5831: Deleting consensus metadata
I20260927 14:00:53.172041 10036 catalog_manager.cc:4841] TS cee81b89a78d4ec2851f1c46ad4972aa (127.22.26.195:36073): tablet 90406ba02c1e4f36b55475e417fe17e1 (table test-table [id=f3db14a9f8004800a8a40b1fed0c5135]) successfully deleted
I20260927 14:00:53.173141 10035 catalog_manager.cc:4841] TS fb138f0dff9c4b0f8c4d3bb7abca5831 (127.22.26.194:35615): tablet 90406ba02c1e4f36b55475e417fe17e1 (table test-table [id=f3db14a9f8004800a8a40b1fed0c5135]) successfully deleted
I20260927 14:00:53.297147 10539 tablet_replica.cc:330] T 90406ba02c1e4f36b55475e417fe17e1 P a0bda8a0500643d8a807e9d10b32d5ea: stopping tablet replica
I20260927 14:00:53.297516 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 10108
I20260927 14:00:53.315358 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 10239
W20260927 14:00:53.315398 10034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.26.193:40701 (error 108)
I20260927 14:00:53.317935 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.317192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40701 (local address 127.0.0.1:42744)
0927 14:00:53.317657 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:53.317784 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":214,"threads_started":1}
W20260927 14:00:53.318209 10034 catalog_manager.cc:4584] TS a0bda8a0500643d8a807e9d10b32d5ea (127.22.26.193:40701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90406ba02c1e4f36b55475e417fe17e1: Network error: Client connection negotiation failed: client connection to 127.22.26.193:40701: connect: Connection refused (error 111)
I20260927 14:00:53.323648 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.316215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.22.26.194:35615)
0927 14:00:53.316401 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:53.316407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:53.316708 (+ 301us) server_negotiation.cc:407] Connection header received
0927 14:00:53.316787 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:53.316793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:53.316912 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:53.317021 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:53.317678 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.318696 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.320112 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.320755 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.320885 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:53.321952 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:53.321963 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:53.321983 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:53.322769 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:53.322836 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:53.322861 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:53.322967 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:53.323161 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 14:00:53.323283 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 14:00:53.324323 10361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:53.371853 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.371308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40701 (local address 127.0.0.1:42754)
0927 14:00:53.371551 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:53.371667 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 14:00:53.373051 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.372656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40701 (local address 127.0.0.1:42770)
0927 14:00:53.372829 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:53.372910 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 14:00:53.415843 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 10239
I20260927 14:00:53.433933 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.433346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40701 (local address 127.0.0.1:42776)
0927 14:00:53.433613 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:53.433741 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 14:00:53.435232 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.434744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40701 (local address 127.0.0.1:42782)
0927 14:00:53.434921 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:53.435011 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 14:00:53.437321 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 10369
I20260927 14:00:53.444423 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.438084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.22.26.195:36073)
0927 14:00:53.438322 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:53.438328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:53.438351 (+ 23us) server_negotiation.cc:407] Connection header received
0927 14:00:53.438467 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:53.438474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:53.438550 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:53.438659 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:53.439066 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.439932 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.441175 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:53.441588 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:53.441720 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:53.442852 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:53.442860 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:53.442873 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:53.443640 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:53.443700 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:53.443724 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:53.443848 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:53.444003 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 14:00:53.444101 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 14:00:53.445113 10491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:53.516542 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.515966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40701 (local address 127.0.0.1:42786)
0927 14:00:53.516239 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:53.516350 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 14:00:53.517755 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.517268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40701 (local address 127.0.0.1:42792)
0927 14:00:53.517510 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:53.517592 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260927 14:00:53.566645 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 10369
I20260927 14:00:53.585731 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 10018
I20260927 14:00:53.586819 10079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:53.697727 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 10018
I20260927 14:00:53.697770 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:53.697101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.193:40701 (local address 127.0.0.1:42808)
0927 14:00:53.697461 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 14:00:53.697614 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.193:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
2026-09-27T14:00:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3071 ms)
[----------] 6 tests from EncryptionPolicyTest (21842 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:00:53 dist-test-slave-gkks krb5kdc[10549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:00:53 dist-test-slave-gkks krb5kdc[10549](info): set up 2 sockets
Sep 27 14:00:53 dist-test-slave-gkks krb5kdc[10549](info): commencing operation
krb5kdc: starting...
W20260927 14:00:54.827950 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:00:54 dist-test-slave-gkks krb5kdc[10549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:00:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:00:55.094758 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:55.439949 10562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:55.440284 10562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:55.440438 10562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:55.447902 10562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:00:55.448043 10562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:55.448161 10562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:55.448293 10562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:00:55.448374 10562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:00:55.458271 10562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:55.459700 10562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:55.461041 10562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:55.473155 10567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:55.473131 10568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:55.473097 10570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:55.473814 10562 server_base.cc:1034] running on GCE node
I20260927 14:00:55.474570 10562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:55.475960 10562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:55.477152 10562 hybrid_clock.cc:648] HybridClock initialized: now 1790517655477098 us; error 52 us; skew 500 ppm
I20260927 14:00:55.477509 10562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:55 dist-test-slave-gkks krb5kdc[10549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:55.480685 10562 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:00:55.482086 10562 webserver.cc:460] Webserver started at http://127.0.0.1:34531/ using document root <none> and password file <none>
I20260927 14:00:55.482795 10562 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:55.482925 10562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:55.483282 10562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:55.486131 10562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "7a443b3e42504225b21ea64313b56c07"
format_stamp: "Formatted at 2026-09-27 14:00:55 on dist-test-slave-gkks"
server_key: "9cc3602668851118401335f5a1a6c8a3"
server_key_iv: "9604ab4831b318649eea6627dcf69523"
server_key_version: "encryptionkey@0"
I20260927 14:00:55.486884 10562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "7a443b3e42504225b21ea64313b56c07"
format_stamp: "Formatted at 2026-09-27 14:00:55 on dist-test-slave-gkks"
server_key: "9cc3602668851118401335f5a1a6c8a3"
server_key_iv: "9604ab4831b318649eea6627dcf69523"
server_key_version: "encryptionkey@0"
I20260927 14:00:55.491916 10562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 14:00:55.495494 10577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:55.496834 10562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 14:00:55.497032 10562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "7a443b3e42504225b21ea64313b56c07"
format_stamp: "Formatted at 2026-09-27 14:00:55 on dist-test-slave-gkks"
server_key: "9cc3602668851118401335f5a1a6c8a3"
server_key_iv: "9604ab4831b318649eea6627dcf69523"
server_key_version: "encryptionkey@0"
I20260927 14:00:55.497238 10562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:55.534600 10562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:55.538858 10562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:55.539216 10562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:55.558631 10562 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40977
I20260927 14:00:55.558667 10628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40977 every 8 connection(s)
I20260927 14:00:55.560089 10562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:00:55.563962 10629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:55.570097 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 10562
I20260927 14:00:55.570317 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:00:55.571204 22635 external_mini_cluster.cc:1428] Setting key b6e94a0c42af3b326a391fdf8b8ce289
I20260927 14:00:55.575266 10629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07: Bootstrap starting.
I20260927 14:00:55.579046 10629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:55.580163 10629 log.cc:824] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:55.583246 10629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07: No bootstrap required, opened a new log
Sep 27 14:00:55 dist-test-slave-gkks krb5kdc[10549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 14:00:55.591765 10629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a443b3e42504225b21ea64313b56c07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260927 14:00:55.592263 10629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:55.592378 10629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a443b3e42504225b21ea64313b56c07, State: Initialized, Role: FOLLOWER
I20260927 14:00:55.592965 10629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a443b3e42504225b21ea64313b56c07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260927 14:00:55.593247 10629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:00:55.593397 10629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:00:55.593598 10629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:55.595345 10632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:55.573225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45179 (local address 127.0.0.1:40977)
0927 14:00:55.574037 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:55.574050 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:55.574091 (+ 41us) server_negotiation.cc:407] Connection header received
0927 14:00:55.576084 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:55.576107 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:55.576543 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:55.576891 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:55.578757 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:55.580161 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:55.581791 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:55.582434 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:55.582558 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:55.587241 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:55.587249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:55.587263 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:55.587297 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:55.590463 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:55.591510 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:55.591516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:55.591522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:55.591713 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:55.592378 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:55.592382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:55.592384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:55.593035 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:55.593328 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:55.593869 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 14:00:55.594201 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":403,"threads_started":1}
I20260927 14:00:55.596230 10629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a443b3e42504225b21ea64313b56c07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260927 14:00:55.596737 10629 leader_election.cc:302] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a443b3e42504225b21ea64313b56c07; no voters:
I20260927 14:00:55.597436 10629 leader_election.cc:288] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:00:55.597579 10634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:55.599274 10634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [term 1 LEADER]: Becoming Leader. State: Replica: 7a443b3e42504225b21ea64313b56c07, State: Running, Role: LEADER
I20260927 14:00:55.599360 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:00:55.599903 10634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a443b3e42504225b21ea64313b56c07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260927 14:00:55.600030 10629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:00:55.603018 10635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a443b3e42504225b21ea64313b56c07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a443b3e42504225b21ea64313b56c07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260927 14:00:55.603001 10636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a443b3e42504225b21ea64313b56c07. Latest consensus state: current_term: 1 leader_uuid: "7a443b3e42504225b21ea64313b56c07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a443b3e42504225b21ea64313b56c07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260927 14:00:55.603477 10635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:55.603556 10636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:55.613829 10640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:00:55.620464 10640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:00:55.638671 10640 catalog_manager.cc:1347] Generated new cluster ID: 9dbce03ee2024323b137796aa53d48ee
I20260927 14:00:55.638797 10640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:00:55.642243 10632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:55.616500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49775 (local address 127.0.0.1:40977)
0927 14:00:55.616672 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:55.616680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:55.621724 (+ 5044us) server_negotiation.cc:407] Connection header received
0927 14:00:55.622070 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:55.622079 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:55.622184 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:55.622277 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:55.624881 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:55.626477 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:55.628894 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:55.629827 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:55.629952 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:55.634202 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:55.634210 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:55.634215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:00:55.634256 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:55.638682 (+ 4426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:55.639805 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:55.639815 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:55.639822 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:55.640015 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:00:55.640641 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:00:55.640646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:00:55.640650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:00:55.641031 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:55.641280 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:55.641715 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 14:00:55.641891 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 14:00:55.655555 10640 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:00:55.656932 10640 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:00:55.665899 10640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a443b3e42504225b21ea64313b56c07: Generated new TSK 0
I20260927 14:00:55.667045 10640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:00:55.673898 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 10562
I20260927 14:00:55.675628 10624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:00:55.768697 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 10562
W20260927 14:00:55.786418 10547 connection.cc:535] client connection to 127.0.0.1:40977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40977): Connection reset by peer (error 104)
2026-09-27T14:00:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2064 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:00:55 dist-test-slave-gkks krb5kdc[10661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:00:55 dist-test-slave-gkks krb5kdc[10661](info): set up 2 sockets
Sep 27 14:00:55 dist-test-slave-gkks krb5kdc[10661](info): commencing operation
krb5kdc: starting...
W20260927 14:00:56.871666 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:00:57 dist-test-slave-gkks krb5kdc[10661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:00:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:00:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:00:57.126001 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33021
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:57.458452 10674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:57.458731 10674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:57.458822 10674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:57.465910 10674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:00:57.466008 10674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:57.466054 10674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:57.466105 10674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:00:57.466140 10674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:00:57.475844 10674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33021
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:57.477025 10674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:57.478329 10674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:57.489750 10679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:57.489832 10680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:57.489871 10682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:57.490626 10674 server_base.cc:1034] running on GCE node
I20260927 14:00:57.491436 10674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:57.492725 10674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:57.493948 10674 hybrid_clock.cc:648] HybridClock initialized: now 1790517657493911 us; error 54 us; skew 500 ppm
I20260927 14:00:57.494282 10674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:57 dist-test-slave-gkks krb5kdc[10661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:57.498039 10674 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:00:57.499567 10674 webserver.cc:460] Webserver started at http://127.0.0.1:35873/ using document root <none> and password file <none>
I20260927 14:00:57.500200 10674 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:57.500301 10674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:57.500545 10674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:57.503492 10674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "258cef8f20104633970f65d26dec2e96"
format_stamp: "Formatted at 2026-09-27 14:00:57 on dist-test-slave-gkks"
server_key: "45d4840833d7cd62d0e7a7b3d14c9d3e"
server_key_iv: "4025dde3333cfbb95d9fd77dbaacc5dc"
server_key_version: "encryptionkey@0"
I20260927 14:00:57.504201 10674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "258cef8f20104633970f65d26dec2e96"
format_stamp: "Formatted at 2026-09-27 14:00:57 on dist-test-slave-gkks"
server_key: "45d4840833d7cd62d0e7a7b3d14c9d3e"
server_key_iv: "4025dde3333cfbb95d9fd77dbaacc5dc"
server_key_version: "encryptionkey@0"
I20260927 14:00:57.509234 10674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 14:00:57.512729 10689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:57.514117 10674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:00:57.514305 10674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "258cef8f20104633970f65d26dec2e96"
format_stamp: "Formatted at 2026-09-27 14:00:57 on dist-test-slave-gkks"
server_key: "45d4840833d7cd62d0e7a7b3d14c9d3e"
server_key_iv: "4025dde3333cfbb95d9fd77dbaacc5dc"
server_key_version: "encryptionkey@0"
I20260927 14:00:57.514492 10674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:57.534657 10674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:57.538838 10674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:57.539062 10674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:57.556558 10674 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33021
I20260927 14:00:57.556596 10740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33021 every 8 connection(s)
I20260927 14:00:57.557929 10674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:00:57.559355 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 10674
I20260927 14:00:57.559532 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:00:57.560012 22635 external_mini_cluster.cc:1428] Setting key 6ffeae2219fde748facd8d99fb66b714
I20260927 14:00:57.562340 10741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:57.572657 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:57.561691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44557 (local address 127.0.0.1:33021)
0927 14:00:57.562543 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:57.562554 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:57.562595 (+ 41us) server_negotiation.cc:407] Connection header received
0927 14:00:57.564211 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:57.564231 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:57.564593 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:57.564998 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:57.565531 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:57.566888 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:57.568242 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:57.568917 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:57.569122 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:57.570417 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:57.570424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:57.570436 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:57.570982 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:57.571059 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:57.571140 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:57.571333 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:57.571594 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 14:00:57.571763 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":349,"threads_started":1}
I20260927 14:00:57.573324 10741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96: Bootstrap starting.
I20260927 14:00:57.576409 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:00:57.576658 10741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:57.577847 10741 log.cc:824] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:57.581110 10741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96: No bootstrap required, opened a new log
I20260927 14:00:57.588356 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:57.581527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43057 (local address 127.0.0.1:33021)
0927 14:00:57.581764 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:57.581773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:57.582172 (+ 399us) server_negotiation.cc:407] Connection header received
0927 14:00:57.582352 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:57.582355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:57.582409 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:57.582490 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:57.583172 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:57.583939 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:57.585812 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:00:57.586245 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:00:57.586340 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:00:57.587599 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:57.587607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:57.587610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:57.587633 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:57.587668 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:57.587686 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:57.587770 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:57.587977 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 14:00:57.588015 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 14:00:57.590728 10741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258cef8f20104633970f65d26dec2e96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33021 } }
I20260927 14:00:57.591157 10741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:57.591285 10741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 258cef8f20104633970f65d26dec2e96, State: Initialized, Role: FOLLOWER
I20260927 14:00:57.592051 10741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258cef8f20104633970f65d26dec2e96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33021 } }
I20260927 14:00:57.592326 10741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:00:57.592490 10741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:00:57.592695 10741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:57.594971 10741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258cef8f20104633970f65d26dec2e96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33021 } }
I20260927 14:00:57.595568 10741 leader_election.cc:302] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 258cef8f20104633970f65d26dec2e96; no voters:
I20260927 14:00:57.596197 10741 leader_election.cc:288] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:00:57.596342 10751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:57.597055 10751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [term 1 LEADER]: Becoming Leader. State: Replica: 258cef8f20104633970f65d26dec2e96, State: Running, Role: LEADER
I20260927 14:00:57.597601 10751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258cef8f20104633970f65d26dec2e96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33021 } }
I20260927 14:00:57.598353 10741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:00:57.600863 10752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "258cef8f20104633970f65d26dec2e96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258cef8f20104633970f65d26dec2e96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33021 } } }
I20260927 14:00:57.601364 10752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:57.600790 10753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 258cef8f20104633970f65d26dec2e96. Latest consensus state: current_term: 1 leader_uuid: "258cef8f20104633970f65d26dec2e96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258cef8f20104633970f65d26dec2e96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33021 } } }
I20260927 14:00:57.602713 10753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:57.604790 10760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:00:57.610297 10760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:00:57.619102 10760 catalog_manager.cc:1347] Generated new cluster ID: 6472556d3dab412196cb3b8b91692d7e
I20260927 14:00:57.619269 10760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:00:57.640324 10760 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:00:57.641665 10760 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:00:57.649325 10760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 258cef8f20104633970f65d26dec2e96: Generated new TSK 0
I20260927 14:00:57.650166 10760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:00:57.662806 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 10674
W20260927 14:00:57.664048 10736 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44557
W20260927 14:00:57.664815 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 10674: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 14:00:57.664999 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 10674
W20260927 14:00:57.682312 10659 connection.cc:535] client connection to 127.0.0.1:33021 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33021): Connection reset by peer (error 104)
2026-09-27T14:00:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1895 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:00:57 dist-test-slave-gkks krb5kdc[10772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:00:57 dist-test-slave-gkks krb5kdc[10772](info): set up 2 sockets
Sep 27 14:00:57 dist-test-slave-gkks krb5kdc[10772](info): commencing operation
krb5kdc: starting...
W20260927 14:00:58.759723 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:00:58 dist-test-slave-gkks krb5kdc[10772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:00:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:00:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:00:59.010390 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:00:59.336314 10785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:00:59.336647 10785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:00:59.336819 10785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:00:59.344017 10785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:00:59.344179 10785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:00:59.344290 10785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:00:59.344399 10785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:00:59.344483 10785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:00:59.354344 10785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39403
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:00:59.355741 10785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:00:59.357045 10785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:00:59.369262 10791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:59.369304 10793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:00:59.369349 10790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:00:59.369904 10785 server_base.cc:1034] running on GCE node
I20260927 14:00:59.370798 10785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:00:59.372216 10785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:00:59.373416 10785 hybrid_clock.cc:648] HybridClock initialized: now 1790517659373374 us; error 32 us; skew 500 ppm
I20260927 14:00:59.373782 10785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:00:59 dist-test-slave-gkks krb5kdc[10772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:00:59.381603 10785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:00:59.383354 10785 webserver.cc:460] Webserver started at http://127.0.0.1:42431/ using document root <none> and password file <none>
I20260927 14:00:59.384084 10785 fs_manager.cc:359] Metadata directory not provided
I20260927 14:00:59.384236 10785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:00:59.384559 10785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:00:59.387637 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "75446288f0fc426998b56ca363d2bb97"
format_stamp: "Formatted at 2026-09-27 14:00:59 on dist-test-slave-gkks"
server_key: "825fb8555ab2c4c1822f5e26afa22317"
server_key_iv: "6d4ed09b5cc3a7fadb42b2a5c743a376"
server_key_version: "encryptionkey@0"
I20260927 14:00:59.388440 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "75446288f0fc426998b56ca363d2bb97"
format_stamp: "Formatted at 2026-09-27 14:00:59 on dist-test-slave-gkks"
server_key: "825fb8555ab2c4c1822f5e26afa22317"
server_key_iv: "6d4ed09b5cc3a7fadb42b2a5c743a376"
server_key_version: "encryptionkey@0"
I20260927 14:00:59.393641 10785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 14:00:59.397074 10800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:00:59.398357 10785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:00:59.398541 10785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "75446288f0fc426998b56ca363d2bb97"
format_stamp: "Formatted at 2026-09-27 14:00:59 on dist-test-slave-gkks"
server_key: "825fb8555ab2c4c1822f5e26afa22317"
server_key_iv: "6d4ed09b5cc3a7fadb42b2a5c743a376"
server_key_version: "encryptionkey@0"
I20260927 14:00:59.398751 10785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:00:59.404388 10785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:00:59.408432 10785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:00:59.408723 10785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:00:59.429790 10785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39403
I20260927 14:00:59.429821 10851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39403 every 8 connection(s)
I20260927 14:00:59.431339 10785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:00:59.435528 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 10785
I20260927 14:00:59.435724 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:00:59.436269 22635 external_mini_cluster.cc:1428] Setting key a875927f7098eeeba805740c8588093d
I20260927 14:00:59.436697 10852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:00:59.447417 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:59.440045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43707 (local address 127.0.0.1:39403)
0927 14:00:59.440972 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:59.440988 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:59.441050 (+ 62us) server_negotiation.cc:407] Connection header received
0927 14:00:59.443355 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:59.443383 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:59.443848 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:59.444299 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:59.445418 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:59.445426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:59.445442 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:59.445460 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:59.445530 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:59.445561 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:59.445788 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:59.445906 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 14:00:59.446111 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":382,"threads_started":1}
I20260927 14:00:59.451867 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:00:59.452186 10852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97: Bootstrap starting.
I20260927 14:00:59.455209 10852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97: Neither blocks nor log segments found. Creating new log.
I20260927 14:00:59.456593 10852 log.cc:824] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97: Log is configured to *not* fsync() on all Append() calls
I20260927 14:00:59.460429 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:00:59.457772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39553 (local address 127.0.0.1:39403)
0927 14:00:59.457960 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 14:00:59.457968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:00:59.458326 (+ 358us) server_negotiation.cc:407] Connection header received
0927 14:00:59.458456 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:00:59.458461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:00:59.458522 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:00:59.458647 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:00:59.459741 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:00:59.459746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:00:59.459750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:00:59.459768 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:00:59.459815 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:00:59.459838 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:00:59.459970 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 14:00:59.460098 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 14:00:59.460161 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 14:00:59.460578 10852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97: No bootstrap required, opened a new log
I20260927 14:00:59.468436 10852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75446288f0fc426998b56ca363d2bb97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } }
I20260927 14:00:59.468838 10852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:00:59.468962 10852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75446288f0fc426998b56ca363d2bb97, State: Initialized, Role: FOLLOWER
I20260927 14:00:59.469630 10852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75446288f0fc426998b56ca363d2bb97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } }
I20260927 14:00:59.469933 10852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:00:59.470098 10852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:00:59.470278 10852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:00:59.472720 10852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75446288f0fc426998b56ca363d2bb97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } }
I20260927 14:00:59.473341 10852 leader_election.cc:302] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75446288f0fc426998b56ca363d2bb97; no voters:
I20260927 14:00:59.474298 10852 leader_election.cc:288] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:00:59.474400 10862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:00:59.477175 10852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:00:59.481386 10862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [term 1 LEADER]: Becoming Leader. State: Replica: 75446288f0fc426998b56ca363d2bb97, State: Running, Role: LEADER
I20260927 14:00:59.482213 10862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75446288f0fc426998b56ca363d2bb97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } }
I20260927 14:00:59.493448 10863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75446288f0fc426998b56ca363d2bb97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75446288f0fc426998b56ca363d2bb97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } } }
I20260927 14:00:59.493458 10864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75446288f0fc426998b56ca363d2bb97. Latest consensus state: current_term: 1 leader_uuid: "75446288f0fc426998b56ca363d2bb97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75446288f0fc426998b56ca363d2bb97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } } }
I20260927 14:00:59.493963 10864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:59.493963 10863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97 [sys.catalog]: This master's current role is: LEADER
I20260927 14:00:59.494879 10877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:00:59.500739 10877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:00:59.510294 10877 catalog_manager.cc:1347] Generated new cluster ID: f111bcb24a154717ae7d6162d6d8a51b
I20260927 14:00:59.510412 10877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:00:59.527627 10877 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:00:59.528141 10877 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:00:59.544564 10877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75446288f0fc426998b56ca363d2bb97: Generated new TSK 0
I20260927 14:00:59.545665 10877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:00:59.553313 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 10785
W20260927 14:00:59.554384 10847 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43707
W20260927 14:00:59.555251 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 10785: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 14:00:59.555440 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 10785
2026-09-27T14:00:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1889 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:00:59 dist-test-slave-gkks krb5kdc[10883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:00:59 dist-test-slave-gkks krb5kdc[10883](info): set up 2 sockets
Sep 27 14:00:59 dist-test-slave-gkks krb5kdc[10883](info): commencing operation
krb5kdc: starting...
W20260927 14:01:00.687858 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:00 dist-test-slave-gkks krb5kdc[10883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:01:00.940367 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:01.266330 10896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:01.266690 10896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:01.266853 10896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:01.274206 10896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:01.274361 10896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:01.274472 10896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:01.274580 10896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:01.274676 10896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:01.284952 10896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:01.286337 10896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:01.287719 10896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:01.299721 10901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:01.299750 10902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:01.299774 10904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:01.300593 10896 server_base.cc:1034] running on GCE node
I20260927 14:01:01.301324 10896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:01.302698 10896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:01.303975 10896 hybrid_clock.cc:648] HybridClock initialized: now 1790517661303925 us; error 55 us; skew 500 ppm
I20260927 14:01:01.304414 10896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:01 dist-test-slave-gkks krb5kdc[10883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:01.307772 10896 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:01:01.309320 10896 webserver.cc:460] Webserver started at http://127.0.0.1:43067/ using document root <none> and password file <none>
I20260927 14:01:01.309985 10896 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:01.310113 10896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:01.310390 10896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:01.313577 10896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "d236caed62b8425e8aa23c6f76422c56"
format_stamp: "Formatted at 2026-09-27 14:01:01 on dist-test-slave-gkks"
server_key: "21f8e44743f5a4c8c2cd649c587ac95c"
server_key_iv: "08fe222298a06b1b81329f5998b346ca"
server_key_version: "encryptionkey@0"
I20260927 14:01:01.314371 10896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "d236caed62b8425e8aa23c6f76422c56"
format_stamp: "Formatted at 2026-09-27 14:01:01 on dist-test-slave-gkks"
server_key: "21f8e44743f5a4c8c2cd649c587ac95c"
server_key_iv: "08fe222298a06b1b81329f5998b346ca"
server_key_version: "encryptionkey@0"
I20260927 14:01:01.319772 10896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 14:01:01.323344 10911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:01.324681 10896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:01:01.324867 10896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "d236caed62b8425e8aa23c6f76422c56"
format_stamp: "Formatted at 2026-09-27 14:01:01 on dist-test-slave-gkks"
server_key: "21f8e44743f5a4c8c2cd649c587ac95c"
server_key_iv: "08fe222298a06b1b81329f5998b346ca"
server_key_version: "encryptionkey@0"
I20260927 14:01:01.325062 10896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:01.345287 10896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:01.349273 10896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:01.349561 10896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:01.368251 10896 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37859
I20260927 14:01:01.368294 10962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37859 every 8 connection(s)
I20260927 14:01:01.369642 10896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:01.373667 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 10896
I20260927 14:01:01.373734 10963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:01.373858 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:01.374405 22635 external_mini_cluster.cc:1428] Setting key 0bd2ce6d69df8ee2e8e74eb67250e376
I20260927 14:01:01.386886 10963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56: Bootstrap starting.
Sep 27 14:01:01 dist-test-slave-gkks krb5kdc[10883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 14:01:01.390128 10963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:01.391388 10963 log.cc:824] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:01.394222 10963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56: No bootstrap required, opened a new log
I20260927 14:01:01.397253 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:01.376278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33157 (local address 127.0.0.1:37859)
0927 14:01:01.377143 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:01.377155 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:01.377196 (+ 41us) server_negotiation.cc:407] Connection header received
0927 14:01:01.378738 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:01.378759 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:01.379196 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:01.379593 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:01.381178 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:01.382439 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:01.383903 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:01.384747 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:01.384972 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:01.389498 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:01.389506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:01.389521 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:01.389562 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:01.392991 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:01.393713 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:01.393730 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:01.393736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:01.393902 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:01.394362 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:01.394369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:01.394373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:01.394902 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:01.395233 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:01.395771 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 14:01:01.396237 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":394,"threads_started":1}
I20260927 14:01:01.400971 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:01.402556 10963 raft_consensus.cc:348] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d236caed62b8425e8aa23c6f76422c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260927 14:01:01.402937 10963 raft_consensus.cc:374] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:01.403045 10963 raft_consensus.cc:729] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d236caed62b8425e8aa23c6f76422c56, State: Initialized, Role: FOLLOWER
I20260927 14:01:01.403784 10963 consensus_queue.cc:260] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d236caed62b8425e8aa23c6f76422c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260927 14:01:01.404033 10963 raft_consensus.cc:388] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:01.404170 10963 raft_consensus.cc:482] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:01.404357 10963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:01.407047 10963 raft_consensus.cc:504] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d236caed62b8425e8aa23c6f76422c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260927 14:01:01.407626 10963 leader_election.cc:302] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d236caed62b8425e8aa23c6f76422c56; no voters:
I20260927 14:01:01.408344 10963 leader_election.cc:288] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:01.408464 10973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:01.411019 10963 sys_catalog.cc:564] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:01.415135 10973 raft_consensus.cc:686] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [term 1 LEADER]: Becoming Leader. State: Replica: d236caed62b8425e8aa23c6f76422c56, State: Running, Role: LEADER
I20260927 14:01:01.416258 10973 consensus_queue.cc:237] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d236caed62b8425e8aa23c6f76422c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260927 14:01:01.419119 10974 sys_catalog.cc:455] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d236caed62b8425e8aa23c6f76422c56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d236caed62b8425e8aa23c6f76422c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } } }
I20260927 14:01:01.419214 10975 sys_catalog.cc:455] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d236caed62b8425e8aa23c6f76422c56. Latest consensus state: current_term: 1 leader_uuid: "d236caed62b8425e8aa23c6f76422c56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d236caed62b8425e8aa23c6f76422c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } } }
I20260927 14:01:01.419687 10974 sys_catalog.cc:458] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:01.423563 10975 sys_catalog.cc:458] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:01.425482 10981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:01.428851 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:01.405523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56963 (local address 127.0.0.1:37859)
0927 14:01:01.405738 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:01.405746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:01.406155 (+ 409us) server_negotiation.cc:407] Connection header received
0927 14:01:01.406351 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:01.406356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:01.406429 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:01.406538 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:01.408078 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:01.409184 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:01.410423 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:01.411013 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:01.411207 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:01.413820 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:01.413825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:01.413828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:01.413857 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:01.418387 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:01.419556 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:01.419565 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:01.419571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:01.419749 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:01.420231 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:01.420239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:01.420245 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:01.425713 (+ 5468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:01.426131 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:01.427820 (+ 1689us) server_negotiation.cc:299] Negotiation successful
0927 14:01:01.428175 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 14:01:01.437491 10981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:01.446931 10981 catalog_manager.cc:1347] Generated new cluster ID: f0e77ebd79364924a056eb56352adb3d
I20260927 14:01:01.447098 10981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:01.467224 10981 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:01.468529 10981 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:01.485514 10981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d236caed62b8425e8aa23c6f76422c56: Generated new TSK 0
I20260927 14:01:01.486539 10981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:01.503664 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 10896
I20260927 14:01:01.504866 10958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:01:01.600044 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 10896
2026-09-27T14:01:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2045 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:01 dist-test-slave-gkks krb5kdc[10995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:01 dist-test-slave-gkks krb5kdc[10995](info): set up 2 sockets
Sep 27 14:01:01 dist-test-slave-gkks krb5kdc[10995](info): commencing operation
krb5kdc: starting...
W20260927 14:01:02.707741 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:02 dist-test-slave-gkks krb5kdc[10995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:01:02.962524 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:03.312917 11008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:03.313292 11008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:03.313458 11008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:03.320855 11008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:03.321009 11008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:03.321130 11008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:03.321237 11008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:03.321321 11008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:03.331441 11008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:03.332885 11008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:03.334271 11008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:03.346513 11014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:03.347023 11016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:03.346586 11013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:03.352649 11008 server_base.cc:1034] running on GCE node
I20260927 14:01:03.353507 11008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:03.355019 11008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:03.356282 11008 hybrid_clock.cc:648] HybridClock initialized: now 1790517663356237 us; error 43 us; skew 500 ppm
I20260927 14:01:03.356642 11008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:03 dist-test-slave-gkks krb5kdc[10995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:03.359995 11008 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:01:03.361671 11008 webserver.cc:460] Webserver started at http://127.0.0.1:44257/ using document root <none> and password file <none>
I20260927 14:01:03.362435 11008 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:03.362586 11008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:03.362905 11008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:03.366027 11008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "e5385a9fab3b4c5398f01a23c42104a2"
format_stamp: "Formatted at 2026-09-27 14:01:03 on dist-test-slave-gkks"
server_key: "2f18804d8930310e9a780097a2fb5339"
server_key_iv: "90a9acfe1813eaa6cf949a4149b4f29f"
server_key_version: "encryptionkey@0"
I20260927 14:01:03.366839 11008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "e5385a9fab3b4c5398f01a23c42104a2"
format_stamp: "Formatted at 2026-09-27 14:01:03 on dist-test-slave-gkks"
server_key: "2f18804d8930310e9a780097a2fb5339"
server_key_iv: "90a9acfe1813eaa6cf949a4149b4f29f"
server_key_version: "encryptionkey@0"
I20260927 14:01:03.372159 11008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 14:01:03.375523 11023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:03.376853 11008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 14:01:03.377048 11008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "e5385a9fab3b4c5398f01a23c42104a2"
format_stamp: "Formatted at 2026-09-27 14:01:03 on dist-test-slave-gkks"
server_key: "2f18804d8930310e9a780097a2fb5339"
server_key_iv: "90a9acfe1813eaa6cf949a4149b4f29f"
server_key_version: "encryptionkey@0"
I20260927 14:01:03.377257 11008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:03.397727 11008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:03.402336 11008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:03.402640 11008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:03.422013 11008 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35817
I20260927 14:01:03.422016 11074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35817 every 8 connection(s)
I20260927 14:01:03.423527 11008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:03.427896 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 11008
I20260927 14:01:03.428087 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:03.428548 11075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:03.428690 22635 external_mini_cluster.cc:1428] Setting key 0532aa67a31a1b24b0522abd88d17913
I20260927 14:01:03.442147 11075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2: Bootstrap starting.
I20260927 14:01:03.442919 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:03.430681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49959 (local address 127.0.0.1:35817)
0927 14:01:03.431558 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:03.431569 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:03.431612 (+ 43us) server_negotiation.cc:407] Connection header received
0927 14:01:03.433170 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:03.433191 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:03.433573 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:03.433970 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:03.434608 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:03.436166 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:03.437545 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:03.438255 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:03.438460 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:03.439878 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:03.439893 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:03.439910 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:03.440815 (+ 905us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:03.440912 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:03.440943 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:03.441076 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:03.441368 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 14:01:03.441630 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":482,"threads_started":1}
I20260927 14:01:03.445462 11075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:03.446614 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:03.446617 11075 log.cc:824] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:03.449594 11075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2: No bootstrap required, opened a new log
I20260927 14:01:03.457942 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:03.451261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39259 (local address 127.0.0.1:35817)
0927 14:01:03.451490 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:03.451498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:03.451954 (+ 456us) server_negotiation.cc:407] Connection header received
0927 14:01:03.452141 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:03.452145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:03.452206 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:03.452292 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:03.452902 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:03.453607 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:03.455516 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:03.455978 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:03.456072 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:03.457181 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:03.457186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:03.457192 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:03.457209 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:03.457249 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:03.457263 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:03.457337 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:03.457497 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 14:01:03.457567 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 14:01:03.459533 11075 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5385a9fab3b4c5398f01a23c42104a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35817 } }
I20260927 14:01:03.459908 11075 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:03.460012 11075 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5385a9fab3b4c5398f01a23c42104a2, State: Initialized, Role: FOLLOWER
I20260927 14:01:03.460542 11075 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5385a9fab3b4c5398f01a23c42104a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35817 } }
I20260927 14:01:03.460775 11075 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:03.460922 11075 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:03.461104 11075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:03.463671 11075 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5385a9fab3b4c5398f01a23c42104a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35817 } }
I20260927 14:01:03.464155 11075 leader_election.cc:302] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5385a9fab3b4c5398f01a23c42104a2; no voters:
I20260927 14:01:03.464679 11075 leader_election.cc:288] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:03.464807 11085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:03.465484 11085 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [term 1 LEADER]: Becoming Leader. State: Replica: e5385a9fab3b4c5398f01a23c42104a2, State: Running, Role: LEADER
I20260927 14:01:03.466064 11085 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5385a9fab3b4c5398f01a23c42104a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35817 } }
I20260927 14:01:03.466797 11075 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:03.468955 11087 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5385a9fab3b4c5398f01a23c42104a2. Latest consensus state: current_term: 1 leader_uuid: "e5385a9fab3b4c5398f01a23c42104a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5385a9fab3b4c5398f01a23c42104a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35817 } } }
I20260927 14:01:03.469458 11087 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:03.469805 11086 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5385a9fab3b4c5398f01a23c42104a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5385a9fab3b4c5398f01a23c42104a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35817 } } }
I20260927 14:01:03.470116 11086 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:03.471825 11094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:03.477094 11094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:03.485575 11094 catalog_manager.cc:1347] Generated new cluster ID: 947f5801107947229ea520b325fa6d76
I20260927 14:01:03.485703 11094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:03.500221 11094 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:03.501447 11094 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:03.509001 11094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5385a9fab3b4c5398f01a23c42104a2: Generated new TSK 0
I20260927 14:01:03.510000 11094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:03.525333 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11008
W20260927 14:01:03.526808 11070 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49959
W20260927 14:01:03.527549 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11008: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 14:01:03.527714 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 11008
2026-09-27T14:01:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:03 dist-test-slave-gkks krb5kdc[11106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:03 dist-test-slave-gkks krb5kdc[11106](info): set up 2 sockets
Sep 27 14:01:03 dist-test-slave-gkks krb5kdc[11106](info): commencing operation
krb5kdc: starting...
W20260927 14:01:04.631780 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:04 dist-test-slave-gkks krb5kdc[11106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:01:04.886508 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:05.218385 11119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:05.218668 11119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:05.218760 11119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:05.225838 11119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:05.225941 11119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:05.225988 11119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:05.226049 11119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:05.226086 11119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:05.235724 11119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:05.236923 11119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:05.238178 11119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:05.249752 11124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:05.249857 11127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:05.249830 11125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:05.250787 11119 server_base.cc:1034] running on GCE node
I20260927 14:01:05.251602 11119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:05.252949 11119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:05.254186 11119 hybrid_clock.cc:648] HybridClock initialized: now 1790517665254142 us; error 55 us; skew 500 ppm
I20260927 14:01:05.254544 11119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:05 dist-test-slave-gkks krb5kdc[11106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:05.267719 11119 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:01:05.269397 11119 webserver.cc:460] Webserver started at http://127.0.0.1:41749/ using document root <none> and password file <none>
I20260927 14:01:05.270129 11119 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:05.270282 11119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:05.270615 11119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:05.273705 11119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "8945f7a630e04180a7542fa0d023737e"
format_stamp: "Formatted at 2026-09-27 14:01:05 on dist-test-slave-gkks"
server_key: "ec1e8c22a80527a9d715a3d17c80d8f1"
server_key_iv: "a3b26b91e0677e25462e5fa5166804c0"
server_key_version: "encryptionkey@0"
I20260927 14:01:05.274509 11119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "8945f7a630e04180a7542fa0d023737e"
format_stamp: "Formatted at 2026-09-27 14:01:05 on dist-test-slave-gkks"
server_key: "ec1e8c22a80527a9d715a3d17c80d8f1"
server_key_iv: "a3b26b91e0677e25462e5fa5166804c0"
server_key_version: "encryptionkey@0"
I20260927 14:01:05.279964 11119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 14:01:05.283574 11134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:05.284998 11119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:01:05.285207 11119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "8945f7a630e04180a7542fa0d023737e"
format_stamp: "Formatted at 2026-09-27 14:01:05 on dist-test-slave-gkks"
server_key: "ec1e8c22a80527a9d715a3d17c80d8f1"
server_key_iv: "a3b26b91e0677e25462e5fa5166804c0"
server_key_version: "encryptionkey@0"
I20260927 14:01:05.285439 11119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:05.291915 11119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:05.296300 11119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:05.296638 11119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:05.316645 11119 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39537
I20260927 14:01:05.316690 11185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39537 every 8 connection(s)
I20260927 14:01:05.318198 11119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:05.322122 11186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:05.322598 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 11119
I20260927 14:01:05.322779 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:05.323374 22635 external_mini_cluster.cc:1428] Setting key c634a608822f0d83fd3f89fb56aaf2db
I20260927 14:01:05.331664 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:05.325288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34555 (local address 127.0.0.1:39537)
0927 14:01:05.326173 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:05.326186 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:05.326230 (+ 44us) server_negotiation.cc:407] Connection header received
0927 14:01:05.328343 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:05.328369 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:05.328712 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:05.329034 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:05.330097 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:05.330104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:05.330117 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:05.330130 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:05.330187 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:05.330209 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:05.330367 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:05.330489 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 14:01:05.330708 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":455,"threads_started":1}
I20260927 14:01:05.335675 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:05.335806 11186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e: Bootstrap starting.
I20260927 14:01:05.339572 11186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:05.340909 11186 log.cc:824] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:05.343494 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:05.341054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43815 (local address 127.0.0.1:39537)
0927 14:01:05.341283 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:05.341289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:05.341654 (+ 365us) server_negotiation.cc:407] Connection header received
0927 14:01:05.341799 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:05.341805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:05.341866 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:05.341986 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:05.342817 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:05.342821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:05.342824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:05.342838 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:05.342873 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:05.342887 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:05.342970 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:05.343155 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 14:01:05.343205 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 14:01:05.344285 11186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e: No bootstrap required, opened a new log
I20260927 14:01:05.351472 11186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8945f7a630e04180a7542fa0d023737e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260927 14:01:05.351804 11186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:05.351873 11186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8945f7a630e04180a7542fa0d023737e, State: Initialized, Role: FOLLOWER
I20260927 14:01:05.352321 11186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8945f7a630e04180a7542fa0d023737e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260927 14:01:05.352515 11186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:05.352612 11186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:05.352723 11186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:05.354926 11186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8945f7a630e04180a7542fa0d023737e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260927 14:01:05.355422 11186 leader_election.cc:302] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8945f7a630e04180a7542fa0d023737e; no voters:
I20260927 14:01:05.355939 11186 leader_election.cc:288] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:05.356091 11196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:05.356864 11196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [term 1 LEADER]: Becoming Leader. State: Replica: 8945f7a630e04180a7542fa0d023737e, State: Running, Role: LEADER
I20260927 14:01:05.357471 11196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8945f7a630e04180a7542fa0d023737e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260927 14:01:05.358050 11186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:05.360500 11198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8945f7a630e04180a7542fa0d023737e. Latest consensus state: current_term: 1 leader_uuid: "8945f7a630e04180a7542fa0d023737e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8945f7a630e04180a7542fa0d023737e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } } }
I20260927 14:01:05.360471 11197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8945f7a630e04180a7542fa0d023737e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8945f7a630e04180a7542fa0d023737e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } } }
I20260927 14:01:05.360942 11198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:05.360975 11197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:05.363481 11204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:05.368520 11204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:05.376833 11204 catalog_manager.cc:1347] Generated new cluster ID: 450cac1c247f410aa3873db3ad9fe5ba
I20260927 14:01:05.376945 11204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:05.394898 11204 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:05.395402 11204 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:05.403306 11204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8945f7a630e04180a7542fa0d023737e: Generated new TSK 0
I20260927 14:01:05.404146 11204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:05.416291 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11119
W20260927 14:01:05.417488 11181 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34555
W20260927 14:01:05.418334 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11119: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 14:01:05.418494 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 11119
2026-09-27T14:01:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:05 dist-test-slave-gkks krb5kdc[11217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:05 dist-test-slave-gkks krb5kdc[11217](info): set up 2 sockets
Sep 27 14:01:05 dist-test-slave-gkks krb5kdc[11217](info): commencing operation
krb5kdc: starting...
W20260927 14:01:06.535775 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:06 dist-test-slave-gkks krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:01:06.784293 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:07.116559 11230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:07.116838 11230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:07.116930 11230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:07.124089 11230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:07.124190 11230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:07.124243 11230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:07.124300 11230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:07.124344 11230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:07.133906 11230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:07.135131 11230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:07.136384 11230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:07.148208 11236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:07.148180 11235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:07.148399 11238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:07.149045 11230 server_base.cc:1034] running on GCE node
I20260927 14:01:07.149874 11230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:07.151311 11230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:07.152514 11230 hybrid_clock.cc:648] HybridClock initialized: now 1790517667152473 us; error 33 us; skew 500 ppm
I20260927 14:01:07.152864 11230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:07 dist-test-slave-gkks krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:07.155884 11230 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:01:07.157330 11230 webserver.cc:460] Webserver started at http://127.0.0.1:43885/ using document root <none> and password file <none>
I20260927 14:01:07.157998 11230 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:07.158128 11230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:07.158427 11230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:07.161350 11230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "c38d044eff4f4347809792a9327db0a1"
format_stamp: "Formatted at 2026-09-27 14:01:07 on dist-test-slave-gkks"
server_key: "25f6d3236982d30a3f782ab40b672804"
server_key_iv: "c0f37233940b33d711f78301199895b5"
server_key_version: "encryptionkey@0"
I20260927 14:01:07.162141 11230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "c38d044eff4f4347809792a9327db0a1"
format_stamp: "Formatted at 2026-09-27 14:01:07 on dist-test-slave-gkks"
server_key: "25f6d3236982d30a3f782ab40b672804"
server_key_iv: "c0f37233940b33d711f78301199895b5"
server_key_version: "encryptionkey@0"
I20260927 14:01:07.167055 11230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 14:01:07.170447 11245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:07.171737 11230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:01:07.171922 11230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "c38d044eff4f4347809792a9327db0a1"
format_stamp: "Formatted at 2026-09-27 14:01:07 on dist-test-slave-gkks"
server_key: "25f6d3236982d30a3f782ab40b672804"
server_key_iv: "c0f37233940b33d711f78301199895b5"
server_key_version: "encryptionkey@0"
I20260927 14:01:07.172119 11230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:07.192863 11230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:07.197664 11230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:07.197973 11230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:07.216905 11230 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35217
I20260927 14:01:07.216945 11296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35217 every 8 connection(s)
I20260927 14:01:07.218303 11230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:07.222183 11297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:07.228719 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 11230
I20260927 14:01:07.228927 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:07.229367 22635 external_mini_cluster.cc:1428] Setting key 0fdcf90943a8f9201552009e214d022e
I20260927 14:01:07.232950 11297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1: Bootstrap starting.
I20260927 14:01:07.236660 11297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:07.237715 11297 log.cc:824] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:07.240792 11297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1: No bootstrap required, opened a new log
Sep 27 14:01:07 dist-test-slave-gkks krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 14:01:07.251608 11297 raft_consensus.cc:348] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c38d044eff4f4347809792a9327db0a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260927 14:01:07.252089 11297 raft_consensus.cc:374] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:07.252210 11297 raft_consensus.cc:729] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c38d044eff4f4347809792a9327db0a1, State: Initialized, Role: FOLLOWER
I20260927 14:01:07.252883 11297 consensus_queue.cc:260] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c38d044eff4f4347809792a9327db0a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260927 14:01:07.253185 11297 raft_consensus.cc:388] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:07.253340 11297 raft_consensus.cc:482] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:07.253517 11297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:07.254076 11300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:07.231252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:33309 (local address 127.0.0.1:35217)
0927 14:01:07.232007 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:07.232018 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:07.232061 (+ 43us) server_negotiation.cc:407] Connection header received
0927 14:01:07.233763 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:07.233786 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:07.234165 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:07.234495 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:07.236430 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:07.237888 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:07.239518 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:07.240180 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:07.240313 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:07.245206 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:07.245216 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:07.245245 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:07.245287 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:07.249363 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:07.250173 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:07.250182 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:07.250191 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:07.250399 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:07.250820 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:07.250825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:07.250828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:07.251550 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:07.251885 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:07.252312 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 14:01:07.252636 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":354,"threads_started":1}
I20260927 14:01:07.256155 11297 raft_consensus.cc:504] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c38d044eff4f4347809792a9327db0a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260927 14:01:07.256704 11297 leader_election.cc:302] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c38d044eff4f4347809792a9327db0a1; no voters:
I20260927 14:01:07.257350 11297 leader_election.cc:288] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:07.257472 11302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:07.258033 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:07.259435 11302 raft_consensus.cc:686] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [term 1 LEADER]: Becoming Leader. State: Replica: c38d044eff4f4347809792a9327db0a1, State: Running, Role: LEADER
I20260927 14:01:07.259598 11297 sys_catalog.cc:564] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:07.259990 11302 consensus_queue.cc:237] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c38d044eff4f4347809792a9327db0a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260927 14:01:07.263074 11304 sys_catalog.cc:455] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c38d044eff4f4347809792a9327db0a1. Latest consensus state: current_term: 1 leader_uuid: "c38d044eff4f4347809792a9327db0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c38d044eff4f4347809792a9327db0a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } } }
I20260927 14:01:07.263548 11304 sys_catalog.cc:458] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:07.263203 11303 sys_catalog.cc:455] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c38d044eff4f4347809792a9327db0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c38d044eff4f4347809792a9327db0a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } } }
I20260927 14:01:07.264148 11303 sys_catalog.cc:458] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:07.266774 11314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:07.278105 11314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:07.292953 11300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:07.273934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:59159 (local address 127.0.0.1:35217)
0927 14:01:07.274158 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:07.274166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:07.275394 (+ 1228us) server_negotiation.cc:407] Connection header received
0927 14:01:07.275609 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:07.275614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:07.275700 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:07.275809 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:07.277667 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:07.278927 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:07.280799 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:07.281515 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:07.281608 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:07.284941 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:07.284946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:07.284950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:07.284980 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:07.289678 (+ 4698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:07.290637 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:07.290645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:07.290651 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:07.290828 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:07.291384 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:07.291397 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:07.291399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:07.291722 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:07.291999 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:07.292410 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 14:01:07.292582 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 14:01:07.293335 11314 catalog_manager.cc:1347] Generated new cluster ID: 715d1b3afdda42b8baf256558b729a0c
I20260927 14:01:07.293445 11314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:07.317227 11314 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:07.318590 11314 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:07.325453 11314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c38d044eff4f4347809792a9327db0a1: Generated new TSK 0
I20260927 14:01:07.326301 11314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:07.335150 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11230
I20260927 14:01:07.336484 11292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:01:07.435225 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 11230
W20260927 14:01:07.453397 11215 connection.cc:535] client connection to 127.0.0.1:35217 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35217): Connection reset by peer (error 104)
2026-09-27T14:01:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2015 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:07 dist-test-slave-gkks krb5kdc[11329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:07 dist-test-slave-gkks krb5kdc[11329](info): set up 2 sockets
Sep 27 14:01:07 dist-test-slave-gkks krb5kdc[11329](info): commencing operation
krb5kdc: starting...
W20260927 14:01:08.547967 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:08 dist-test-slave-gkks krb5kdc[11329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:01:08.809962 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:09.141196 11342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:09.141500 11342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:09.141681 11342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:09.149237 11342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:09.149365 11342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:09.149469 11342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:09.149552 11342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:09.149591 11342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:09.159410 11342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:09.160706 11342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:09.161994 11342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:09.174011 11347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:09.174101 11348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:09.174008 11350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:09.174847 11342 server_base.cc:1034] running on GCE node
I20260927 14:01:09.175633 11342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:09.177032 11342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:09.178254 11342 hybrid_clock.cc:648] HybridClock initialized: now 1790517669178212 us; error 35 us; skew 500 ppm
I20260927 14:01:09.178624 11342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:09 dist-test-slave-gkks krb5kdc[11329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:09.181784 11342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:01:09.183192 11342 webserver.cc:460] Webserver started at http://127.0.0.1:44269/ using document root <none> and password file <none>
I20260927 14:01:09.183835 11342 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:09.183933 11342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:09.184176 11342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:09.187220 11342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "be94acb6cfb54e54b3259bac89f26012"
format_stamp: "Formatted at 2026-09-27 14:01:09 on dist-test-slave-gkks"
server_key: "7f9985d8063d82135a514e6a52c65361"
server_key_iv: "14d2a893bfa0cac3275f369654bd06df"
server_key_version: "encryptionkey@0"
I20260927 14:01:09.188005 11342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "be94acb6cfb54e54b3259bac89f26012"
format_stamp: "Formatted at 2026-09-27 14:01:09 on dist-test-slave-gkks"
server_key: "7f9985d8063d82135a514e6a52c65361"
server_key_iv: "14d2a893bfa0cac3275f369654bd06df"
server_key_version: "encryptionkey@0"
I20260927 14:01:09.192991 11342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 14:01:09.196321 11357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:09.197566 11342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:01:09.197741 11342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "be94acb6cfb54e54b3259bac89f26012"
format_stamp: "Formatted at 2026-09-27 14:01:09 on dist-test-slave-gkks"
server_key: "7f9985d8063d82135a514e6a52c65361"
server_key_iv: "14d2a893bfa0cac3275f369654bd06df"
server_key_version: "encryptionkey@0"
I20260927 14:01:09.197945 11342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:09.216444 11342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:09.220434 11342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:09.220726 11342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:09.239879 11342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34989
I20260927 14:01:09.239917 11408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34989 every 8 connection(s)
I20260927 14:01:09.241367 11342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:09.245334 11409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:09.246037 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 11342
I20260927 14:01:09.246256 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:09.246737 22635 external_mini_cluster.cc:1428] Setting key 55b3aff22c17a839707b644078ec794b
I20260927 14:01:09.259675 11409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012: Bootstrap starting.
I20260927 14:01:09.261337 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:09.248717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:39349 (local address 127.0.0.1:34989)
0927 14:01:09.249634 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:09.249647 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:09.249690 (+ 43us) server_negotiation.cc:407] Connection header received
0927 14:01:09.251762 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:09.251797 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:09.252294 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:09.252652 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:09.253320 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:09.254602 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:09.256622 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:09.257397 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:09.257536 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:09.258890 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:09.258898 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:09.258913 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:09.259594 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:09.259680 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:09.259708 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:09.259893 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:09.260193 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 14:01:09.260362 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":464,"threads_started":1}
I20260927 14:01:09.263348 11409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:09.264544 11409 log.cc:824] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:09.265679 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:09.267509 11409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012: No bootstrap required, opened a new log
I20260927 14:01:09.276474 11409 raft_consensus.cc:348] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be94acb6cfb54e54b3259bac89f26012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34989 } }
I20260927 14:01:09.276988 11409 raft_consensus.cc:374] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:09.277104 11409 raft_consensus.cc:729] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be94acb6cfb54e54b3259bac89f26012, State: Initialized, Role: FOLLOWER
I20260927 14:01:09.277695 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:09.270205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:59011 (local address 127.0.0.1:34989)
0927 14:01:09.270399 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:09.270406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:09.270995 (+ 589us) server_negotiation.cc:407] Connection header received
0927 14:01:09.271225 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:09.271231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:09.271324 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:09.271436 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:09.272114 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:09.273314 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:09.274735 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:09.275485 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:09.275612 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:09.276808 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:09.276815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:09.276820 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:09.276843 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:09.276908 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:09.276930 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:09.277063 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:09.277327 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 14:01:09.277381 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 14:01:09.277837 11409 consensus_queue.cc:260] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be94acb6cfb54e54b3259bac89f26012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34989 } }
I20260927 14:01:09.278103 11409 raft_consensus.cc:388] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:09.278273 11409 raft_consensus.cc:482] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:09.278456 11409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:09.280786 11409 raft_consensus.cc:504] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be94acb6cfb54e54b3259bac89f26012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34989 } }
I20260927 14:01:09.281283 11409 leader_election.cc:302] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be94acb6cfb54e54b3259bac89f26012; no voters:
I20260927 14:01:09.281834 11409 leader_election.cc:288] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:09.281971 11419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:09.283015 11419 raft_consensus.cc:686] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [term 1 LEADER]: Becoming Leader. State: Replica: be94acb6cfb54e54b3259bac89f26012, State: Running, Role: LEADER
I20260927 14:01:09.283723 11419 consensus_queue.cc:237] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be94acb6cfb54e54b3259bac89f26012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34989 } }
I20260927 14:01:09.284173 11409 sys_catalog.cc:564] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:09.286665 11420 sys_catalog.cc:455] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be94acb6cfb54e54b3259bac89f26012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be94acb6cfb54e54b3259bac89f26012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34989 } } }
I20260927 14:01:09.287113 11420 sys_catalog.cc:458] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:09.287549 11421 sys_catalog.cc:455] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be94acb6cfb54e54b3259bac89f26012. Latest consensus state: current_term: 1 leader_uuid: "be94acb6cfb54e54b3259bac89f26012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be94acb6cfb54e54b3259bac89f26012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34989 } } }
I20260927 14:01:09.287838 11421 sys_catalog.cc:458] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:09.288784 11428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:09.298481 11428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:09.308044 11428 catalog_manager.cc:1347] Generated new cluster ID: dfa2c08b916f471eb4bf3c6f2834799c
I20260927 14:01:09.308179 11428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:09.319011 11428 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:09.320351 11428 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:09.330194 11428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be94acb6cfb54e54b3259bac89f26012: Generated new TSK 0
I20260927 14:01:09.331022 11428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:09.347175 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11342
W20260927 14:01:09.348559 11404 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.58:39349
W20260927 14:01:09.349413 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11342: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 14:01:09.349576 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 11342
W20260927 14:01:09.366688 11327 connection.cc:535] client connection to 127.0.0.1:34989 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34989): Connection reset by peer (error 104)
2026-09-27T14:01:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:09 dist-test-slave-gkks krb5kdc[11440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:09 dist-test-slave-gkks krb5kdc[11440](info): set up 2 sockets
Sep 27 14:01:09 dist-test-slave-gkks krb5kdc[11440](info): commencing operation
krb5kdc: starting...
W20260927 14:01:10.455735 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:10 dist-test-slave-gkks krb5kdc[11440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:01:10.714465 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33445
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:11.040941 11453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:11.041216 11453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:11.041388 11453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:11.048657 11453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:11.048767 11453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:11.048873 11453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:11.048976 11453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:11.049041 11453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:11.058907 11453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:11.060177 11453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:11.061453 11453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:11.073128 11461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:11.073208 11458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:11.073241 11459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:11.074178 11453 server_base.cc:1034] running on GCE node
I20260927 14:01:11.074932 11453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:11.076367 11453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:11.077593 11453 hybrid_clock.cc:648] HybridClock initialized: now 1790517671077551 us; error 37 us; skew 500 ppm
I20260927 14:01:11.077936 11453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:11 dist-test-slave-gkks krb5kdc[11440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:11.087751 11453 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:01:11.089329 11453 webserver.cc:460] Webserver started at http://127.0.0.1:40003/ using document root <none> and password file <none>
I20260927 14:01:11.090224 11453 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:11.090379 11453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:11.090691 11453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:11.093763 11453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "4ebe2626e6bb423d9f35bbbfa0195f1a"
format_stamp: "Formatted at 2026-09-27 14:01:11 on dist-test-slave-gkks"
server_key: "74c558f72760d8dca08365246a08d87b"
server_key_iv: "ab78192e78b20b6cd44ac57f180162fb"
server_key_version: "encryptionkey@0"
I20260927 14:01:11.094527 11453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "4ebe2626e6bb423d9f35bbbfa0195f1a"
format_stamp: "Formatted at 2026-09-27 14:01:11 on dist-test-slave-gkks"
server_key: "74c558f72760d8dca08365246a08d87b"
server_key_iv: "ab78192e78b20b6cd44ac57f180162fb"
server_key_version: "encryptionkey@0"
I20260927 14:01:11.099669 11453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 14:01:11.103242 11468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:11.104475 11453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:01:11.104646 11453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "4ebe2626e6bb423d9f35bbbfa0195f1a"
format_stamp: "Formatted at 2026-09-27 14:01:11 on dist-test-slave-gkks"
server_key: "74c558f72760d8dca08365246a08d87b"
server_key_iv: "ab78192e78b20b6cd44ac57f180162fb"
server_key_version: "encryptionkey@0"
I20260927 14:01:11.104832 11453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:11.110563 11453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:11.114825 11453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:11.115164 11453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:11.132903 11453 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33445
I20260927 14:01:11.132980 11519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33445 every 8 connection(s)
I20260927 14:01:11.134254 11453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:11.137882 11520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:11.138144 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 11453
I20260927 14:01:11.138335 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:11.138896 22635 external_mini_cluster.cc:1428] Setting key 5eef72dd0d4af2f68aa94f0e4022f251
I20260927 14:01:11.149837 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:11.140908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:32965 (local address 127.0.0.1:33445)
0927 14:01:11.141870 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:11.141886 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:11.143666 (+ 1780us) server_negotiation.cc:407] Connection header received
0927 14:01:11.145961 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:11.145988 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:11.146420 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:11.146890 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:11.147907 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:11.147916 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:11.147932 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:11.147948 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:11.148026 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:11.148059 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:11.148309 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:11.148438 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 14:01:11.148640 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":380,"threads_started":1}
I20260927 14:01:11.151213 11520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a: Bootstrap starting.
I20260927 14:01:11.153640 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:11.154729 11520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:11.155805 11520 log.cc:824] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:11.158974 11520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a: No bootstrap required, opened a new log
I20260927 14:01:11.160743 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:11.158003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:41819 (local address 127.0.0.1:33445)
0927 14:01:11.158184 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:11.158190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:11.158752 (+ 562us) server_negotiation.cc:407] Connection header received
0927 14:01:11.158937 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:11.158941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:11.158989 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:11.159116 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:11.160009 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:11.160015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:11.160019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:11.160043 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:11.160078 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:11.160093 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:11.160208 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:11.160344 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 14:01:11.160415 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 14:01:11.166947 11520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebe2626e6bb423d9f35bbbfa0195f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33445 } }
I20260927 14:01:11.167478 11520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:11.167586 11520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebe2626e6bb423d9f35bbbfa0195f1a, State: Initialized, Role: FOLLOWER
I20260927 14:01:11.168151 11520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebe2626e6bb423d9f35bbbfa0195f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33445 } }
I20260927 14:01:11.168424 11520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:11.168567 11520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:11.168740 11520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:11.170964 11520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebe2626e6bb423d9f35bbbfa0195f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33445 } }
I20260927 14:01:11.171567 11520 leader_election.cc:302] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ebe2626e6bb423d9f35bbbfa0195f1a; no voters:
I20260927 14:01:11.172286 11520 leader_election.cc:288] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:11.172394 11530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:11.173022 11530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [term 1 LEADER]: Becoming Leader. State: Replica: 4ebe2626e6bb423d9f35bbbfa0195f1a, State: Running, Role: LEADER
I20260927 14:01:11.173566 11530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebe2626e6bb423d9f35bbbfa0195f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33445 } }
I20260927 14:01:11.174607 11520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:11.176535 11532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ebe2626e6bb423d9f35bbbfa0195f1a. Latest consensus state: current_term: 1 leader_uuid: "4ebe2626e6bb423d9f35bbbfa0195f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebe2626e6bb423d9f35bbbfa0195f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33445 } } }
I20260927 14:01:11.177026 11532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:11.177714 11531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ebe2626e6bb423d9f35bbbfa0195f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebe2626e6bb423d9f35bbbfa0195f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33445 } } }
I20260927 14:01:11.178018 11531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:11.178602 11535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:11.183866 11535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:11.193132 11535 catalog_manager.cc:1347] Generated new cluster ID: 5114a21ec6e448ff87e8432206525e38
I20260927 14:01:11.193235 11535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:11.229606 11535 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:11.230052 11535 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:11.241724 11535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ebe2626e6bb423d9f35bbbfa0195f1a: Generated new TSK 0
I20260927 14:01:11.242733 11535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:11.256623 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11453
W20260927 14:01:11.257769 11515 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.58:32965
W20260927 14:01:11.258522 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11453: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 14:01:11.258683 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 11453
2026-09-27T14:01:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1905 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:11 dist-test-slave-gkks krb5kdc[11551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:11 dist-test-slave-gkks krb5kdc[11551](info): set up 2 sockets
Sep 27 14:01:11 dist-test-slave-gkks krb5kdc[11551](info): commencing operation
krb5kdc: starting...
W20260927 14:01:12.367643 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:12 dist-test-slave-gkks krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:01:12.612993 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:12.943010 11564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:12.943441 11564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:12.943634 11564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:12.950927 11564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:12.951045 11564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:12.951196 11564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:12.951275 11564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:12.951326 11564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:12.961354 11564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32903
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:12.962606 11564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:12.963964 11564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:12.976123 11572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:12.976182 11569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:12.976212 11570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:12.977029 11564 server_base.cc:1034] running on GCE node
I20260927 14:01:12.977698 11564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:12.979019 11564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:12.980258 11564 hybrid_clock.cc:648] HybridClock initialized: now 1790517672980199 us; error 53 us; skew 500 ppm
I20260927 14:01:12.980603 11564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:12 dist-test-slave-gkks krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:12.983764 11564 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:01:12.985178 11564 webserver.cc:460] Webserver started at http://127.0.0.1:34553/ using document root <none> and password file <none>
I20260927 14:01:12.985833 11564 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:12.985965 11564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:12.986258 11564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:12.989229 11564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "a7e58c0ec6df43d0ac50967160cdb496"
format_stamp: "Formatted at 2026-09-27 14:01:12 on dist-test-slave-gkks"
server_key: "8978da2d863e515e0f0c70e466bbfa08"
server_key_iv: "f13772ddfa12f47776a7ac433a2b4d63"
server_key_version: "encryptionkey@0"
I20260927 14:01:12.990018 11564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "a7e58c0ec6df43d0ac50967160cdb496"
format_stamp: "Formatted at 2026-09-27 14:01:12 on dist-test-slave-gkks"
server_key: "8978da2d863e515e0f0c70e466bbfa08"
server_key_iv: "f13772ddfa12f47776a7ac433a2b4d63"
server_key_version: "encryptionkey@0"
I20260927 14:01:12.995065 11564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 14:01:12.998377 11579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:12.999681 11564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 14:01:12.999877 11564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "a7e58c0ec6df43d0ac50967160cdb496"
format_stamp: "Formatted at 2026-09-27 14:01:12 on dist-test-slave-gkks"
server_key: "8978da2d863e515e0f0c70e466bbfa08"
server_key_iv: "f13772ddfa12f47776a7ac433a2b4d63"
server_key_version: "encryptionkey@0"
I20260927 14:01:13.000077 11564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:13.030054 11564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:13.034554 11564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:13.034842 11564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:13.052644 11564 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32903
I20260927 14:01:13.052676 11630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32903 every 8 connection(s)
I20260927 14:01:13.053951 11564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:13.055619 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 11564
I20260927 14:01:13.055833 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:13.056272 22635 external_mini_cluster.cc:1428] Setting key a352f007ac147b7425265ace4c91d022
I20260927 14:01:13.058101 11631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 14:01:13 dist-test-slave-gkks krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 14:01:13.071213 11631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496: Bootstrap starting.
I20260927 14:01:13.075400 11631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:13.076678 11631 log.cc:824] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:13.080168 11631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496: No bootstrap required, opened a new log
I20260927 14:01:13.080696 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:13.058313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:47165 (local address 127.0.0.1:32903)
0927 14:01:13.059100 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:13.059117 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:13.059164 (+ 47us) server_negotiation.cc:407] Connection header received
0927 14:01:13.061148 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:13.061173 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:13.061669 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:13.062095 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:13.063452 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:13.065431 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:13.066662 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:13.067714 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:13.067915 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:13.072039 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:13.072049 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:13.072066 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:13.072112 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:13.076090 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:13.076936 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:13.076945 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:13.076952 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:13.077153 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:13.077718 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:13.077721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:13.077723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:13.078318 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:13.078605 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:13.079042 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 14:01:13.079413 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":294,"threads_started":1}
I20260927 14:01:13.084693 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:13.088951 11631 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e58c0ec6df43d0ac50967160cdb496" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260927 14:01:13.089394 11631 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:13.089510 11631 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7e58c0ec6df43d0ac50967160cdb496, State: Initialized, Role: FOLLOWER
I20260927 14:01:13.090443 11631 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e58c0ec6df43d0ac50967160cdb496" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260927 14:01:13.090682 11631 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:13.090818 11631 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:13.091001 11631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:13.094092 11631 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e58c0ec6df43d0ac50967160cdb496" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260927 14:01:13.094676 11631 leader_election.cc:302] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7e58c0ec6df43d0ac50967160cdb496; no voters:
I20260927 14:01:13.095402 11631 leader_election.cc:288] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:13.095563 11641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:13.097530 11631 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:13.099728 11641 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [term 1 LEADER]: Becoming Leader. State: Replica: a7e58c0ec6df43d0ac50967160cdb496, State: Running, Role: LEADER
I20260927 14:01:13.100483 11641 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e58c0ec6df43d0ac50967160cdb496" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260927 14:01:13.104080 11643 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7e58c0ec6df43d0ac50967160cdb496. Latest consensus state: current_term: 1 leader_uuid: "a7e58c0ec6df43d0ac50967160cdb496" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e58c0ec6df43d0ac50967160cdb496" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } } }
I20260927 14:01:13.104493 11643 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:13.105029 11642 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7e58c0ec6df43d0ac50967160cdb496" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e58c0ec6df43d0ac50967160cdb496" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } } }
I20260927 14:01:13.105386 11642 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:13.109323 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:13.089064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:60799 (local address 127.0.0.1:32903)
0927 14:01:13.089327 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:13.089336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:13.089866 (+ 530us) server_negotiation.cc:407] Connection header received
0927 14:01:13.090060 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:13.090065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:13.090143 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:13.090243 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:13.091850 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:13.093038 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:13.094278 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:13.094927 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:13.095072 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:13.099444 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:13.099459 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:13.099465 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:13.099508 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:13.104027 (+ 4519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:13.105211 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:13.105216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:13.105220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:13.105432 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:13.106130 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:13.106138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:13.106144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:13.106613 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:13.108387 (+ 1774us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:13.108600 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 14:01:13.108824 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 14:01:13.112944 11653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:13.120573 11653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:13.134232 11653 catalog_manager.cc:1347] Generated new cluster ID: 2b921fa9951145f0a11ef5943e7cead8
I20260927 14:01:13.134368 11653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:13.159363 11653 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:13.161522 11653 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:13.179037 11653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7e58c0ec6df43d0ac50967160cdb496: Generated new TSK 0
I20260927 14:01:13.179992 11653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:13.197652 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11564
I20260927 14:01:13.198868 11626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:01:13.294095 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 11564
W20260927 14:01:13.311540 11549 connection.cc:535] client connection to 127.0.0.1:32903 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32903): Connection reset by peer (error 104)
2026-09-27T14:01:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2039 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:13 dist-test-slave-gkks krb5kdc[11663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:13 dist-test-slave-gkks krb5kdc[11663](info): set up 2 sockets
Sep 27 14:01:13 dist-test-slave-gkks krb5kdc[11663](info): commencing operation
krb5kdc: starting...
W20260927 14:01:14.403898 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:14 dist-test-slave-gkks krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:01:14.662951 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:14.991427 11676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:14.991782 11676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:14.991954 11676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:14.999154 11676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:14.999300 11676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:14.999408 11676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:14.999516 11676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:14.999611 11676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:15.009564 11676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:15.010994 11676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:15.012390 11676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:15.024574 11681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:15.024641 11682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:15.024616 11684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:15.025758 11676 server_base.cc:1034] running on GCE node
I20260927 14:01:15.026592 11676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:15.028101 11676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:15.029315 11676 hybrid_clock.cc:648] HybridClock initialized: now 1790517675029253 us; error 53 us; skew 500 ppm
I20260927 14:01:15.029701 11676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:15 dist-test-slave-gkks krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:15.033138 11676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:01:15.034634 11676 webserver.cc:460] Webserver started at http://127.0.0.1:41869/ using document root <none> and password file <none>
I20260927 14:01:15.035501 11676 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:15.035643 11676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:15.035959 11676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:15.038956 11676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "4fe2ced008c64afbac860fa332655698"
format_stamp: "Formatted at 2026-09-27 14:01:15 on dist-test-slave-gkks"
server_key: "5531abba1abe677cfd477f013b7da857"
server_key_iv: "97631c753df7e115d29fe42b9374a25d"
server_key_version: "encryptionkey@0"
I20260927 14:01:15.039821 11676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "4fe2ced008c64afbac860fa332655698"
format_stamp: "Formatted at 2026-09-27 14:01:15 on dist-test-slave-gkks"
server_key: "5531abba1abe677cfd477f013b7da857"
server_key_iv: "97631c753df7e115d29fe42b9374a25d"
server_key_version: "encryptionkey@0"
I20260927 14:01:15.045169 11676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 14:01:15.048705 11691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:15.050168 11676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 14:01:15.050350 11676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "4fe2ced008c64afbac860fa332655698"
format_stamp: "Formatted at 2026-09-27 14:01:15 on dist-test-slave-gkks"
server_key: "5531abba1abe677cfd477f013b7da857"
server_key_iv: "97631c753df7e115d29fe42b9374a25d"
server_key_version: "encryptionkey@0"
I20260927 14:01:15.050532 11676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:15.064198 11676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:15.068480 11676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:15.068756 11676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:15.086861 11676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35299
I20260927 14:01:15.086941 11742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35299 every 8 connection(s)
I20260927 14:01:15.088325 11676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:15.091905 11743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:15.096587 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 11676
I20260927 14:01:15.096781 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:15.097309 22635 external_mini_cluster.cc:1428] Setting key 7f1b819030944d56d76d552b1157827d
I20260927 14:01:15.103166 11743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698: Bootstrap starting.
I20260927 14:01:15.106557 11743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:15.107784 11743 log.cc:824] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:15.110777 11743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698: No bootstrap required, opened a new log
I20260927 14:01:15.110852 11746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:15.099063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:52513 (local address 127.0.0.1:35299)
0927 14:01:15.099873 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:15.099885 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:15.099929 (+ 44us) server_negotiation.cc:407] Connection header received
0927 14:01:15.101651 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:15.101671 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:15.102022 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:15.102333 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:15.102929 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:15.104339 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:15.106201 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:15.106932 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:15.107054 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:15.108395 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:15.108409 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:15.108432 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:15.109241 (+ 809us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:15.109310 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:15.109339 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:15.109455 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:15.109748 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 14:01:15.109938 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":349,"threads_started":1}
I20260927 14:01:15.114578 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:15.118819 11743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe2ced008c64afbac860fa332655698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35299 } }
I20260927 14:01:15.119297 11743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:15.119410 11743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe2ced008c64afbac860fa332655698, State: Initialized, Role: FOLLOWER
I20260927 14:01:15.120021 11743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe2ced008c64afbac860fa332655698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35299 } }
I20260927 14:01:15.120263 11743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:15.120405 11743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:15.120604 11743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:15.122776 11743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe2ced008c64afbac860fa332655698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35299 } }
I20260927 14:01:15.123319 11743 leader_election.cc:302] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fe2ced008c64afbac860fa332655698; no voters:
I20260927 14:01:15.123960 11743 leader_election.cc:288] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:15.124042 11753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:15.124681 11753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [term 1 LEADER]: Becoming Leader. State: Replica: 4fe2ced008c64afbac860fa332655698, State: Running, Role: LEADER
I20260927 14:01:15.125664 11746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:15.118882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:36551 (local address 127.0.0.1:35299)
0927 14:01:15.119056 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:15.119061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:15.119536 (+ 475us) server_negotiation.cc:407] Connection header received
0927 14:01:15.119733 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:15.119741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:15.119801 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:15.119901 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:15.120459 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:15.121302 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:15.122555 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:15.123133 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:15.123268 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:15.124456 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:15.124464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:15.124470 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:15.124498 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:15.124554 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:15.124588 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:15.124732 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:15.125222 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 14:01:15.125275 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 14:01:15.125458 11753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe2ced008c64afbac860fa332655698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35299 } }
I20260927 14:01:15.127725 11753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe2ced008c64afbac860fa332655698. Latest consensus state: current_term: 1 leader_uuid: "4fe2ced008c64afbac860fa332655698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe2ced008c64afbac860fa332655698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35299 } } }
I20260927 14:01:15.128152 11753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:15.128575 11755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fe2ced008c64afbac860fa332655698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe2ced008c64afbac860fa332655698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35299 } } }
I20260927 14:01:15.128937 11755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:15.132373 11743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:15.132570 11757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:15.138444 11757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:15.146289 11757 catalog_manager.cc:1347] Generated new cluster ID: 0f61eb4488b1480a9ef76782a1018293
I20260927 14:01:15.146394 11757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:15.175505 11757 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:15.177314 11757 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:15.186544 11757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fe2ced008c64afbac860fa332655698: Generated new TSK 0
I20260927 14:01:15.187493 11757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:15.193816 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11676
W20260927 14:01:15.194978 11738 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.58:52513
W20260927 14:01:15.195812 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 14:01:15.196069 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 11676
W20260927 14:01:15.213156 11661 connection.cc:535] client connection to 127.0.0.1:35299 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35299): Connection reset by peer (error 104)
2026-09-27T14:01:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1900 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:15 dist-test-slave-gkks krb5kdc[11774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:15 dist-test-slave-gkks krb5kdc[11774](info): set up 2 sockets
Sep 27 14:01:15 dist-test-slave-gkks krb5kdc[11774](info): commencing operation
krb5kdc: starting...
W20260927 14:01:16.311789 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:16 dist-test-slave-gkks krb5kdc[11774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:01:16.560119 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:16.893365 11787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:16.893671 11787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:16.893778 11787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:16.901109 11787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:16.901229 11787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:16.901280 11787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:16.901342 11787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:16.901389 11787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:16.911386 11787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43607
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:16.912804 11787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:16.914096 11787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:16.926556 11792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:16.926520 11795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:16.926641 11793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:16.928002 11787 server_base.cc:1034] running on GCE node
I20260927 14:01:16.928789 11787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:16.930310 11787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:16.931548 11787 hybrid_clock.cc:648] HybridClock initialized: now 1790517676931505 us; error 54 us; skew 500 ppm
I20260927 14:01:16.931931 11787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:16 dist-test-slave-gkks krb5kdc[11774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:16.942183 11787 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:01:16.944000 11787 webserver.cc:460] Webserver started at http://127.0.0.1:36819/ using document root <none> and password file <none>
I20260927 14:01:16.944759 11787 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:16.944916 11787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:16.945245 11787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:16.948400 11787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "0eb945ba8b2d42e0bd50b0e21b9bc761"
format_stamp: "Formatted at 2026-09-27 14:01:16 on dist-test-slave-gkks"
server_key: "8428b581de4142d05f9de8e831f6803a"
server_key_iv: "332f21f9ca31edbe432053dbaf6dff8b"
server_key_version: "encryptionkey@0"
I20260927 14:01:16.949190 11787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "0eb945ba8b2d42e0bd50b0e21b9bc761"
format_stamp: "Formatted at 2026-09-27 14:01:16 on dist-test-slave-gkks"
server_key: "8428b581de4142d05f9de8e831f6803a"
server_key_iv: "332f21f9ca31edbe432053dbaf6dff8b"
server_key_version: "encryptionkey@0"
I20260927 14:01:16.954507 11787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 14:01:16.958316 11802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:16.959863 11787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:01:16.960062 11787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "0eb945ba8b2d42e0bd50b0e21b9bc761"
format_stamp: "Formatted at 2026-09-27 14:01:16 on dist-test-slave-gkks"
server_key: "8428b581de4142d05f9de8e831f6803a"
server_key_iv: "332f21f9ca31edbe432053dbaf6dff8b"
server_key_version: "encryptionkey@0"
I20260927 14:01:16.960274 11787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:16.966622 11787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:16.971385 11787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:16.971683 11787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:16.990573 11787 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43607
I20260927 14:01:16.990657 11853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43607 every 8 connection(s)
I20260927 14:01:16.992059 11787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:16.993445 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 11787
I20260927 14:01:16.993626 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:16.994163 22635 external_mini_cluster.cc:1428] Setting key ae029fabf46b68fa75b7c2c21bdcaa10
I20260927 14:01:16.996598 11854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:17.001768 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:16.995951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:55949 (local address 127.0.0.1:43607)
0927 14:01:16.996781 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:16.996795 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:16.996856 (+ 61us) server_negotiation.cc:407] Connection header received
0927 14:01:16.998408 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:16.998428 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:16.998763 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:16.999140 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:17.000201 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:17.000207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:17.000218 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:17.000232 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:17.000326 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:17.000345 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:17.000510 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:17.000629 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 14:01:17.000832 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":353,"threads_started":1}
I20260927 14:01:17.005349 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:17.008695 11854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761: Bootstrap starting.
I20260927 14:01:17.012784 11854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:17.013478 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:17.010847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:49493 (local address 127.0.0.1:43607)
0927 14:01:17.011115 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:17.011122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:17.011461 (+ 339us) server_negotiation.cc:407] Connection header received
0927 14:01:17.011659 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:17.011663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:17.011709 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:17.011792 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:17.012753 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:17.012757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:17.012760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:17.012778 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:17.012818 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:17.012833 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:17.012969 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:17.013157 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 14:01:17.013225 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 14:01:17.014045 11854 log.cc:824] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:17.016736 11854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761: No bootstrap required, opened a new log
I20260927 14:01:17.023811 11854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb945ba8b2d42e0bd50b0e21b9bc761" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260927 14:01:17.024169 11854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:17.024278 11854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb945ba8b2d42e0bd50b0e21b9bc761, State: Initialized, Role: FOLLOWER
I20260927 14:01:17.024823 11854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb945ba8b2d42e0bd50b0e21b9bc761" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260927 14:01:17.025056 11854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:17.025199 11854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:17.025368 11854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:17.027516 11854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb945ba8b2d42e0bd50b0e21b9bc761" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260927 14:01:17.028009 11854 leader_election.cc:302] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eb945ba8b2d42e0bd50b0e21b9bc761; no voters:
I20260927 14:01:17.028606 11854 leader_election.cc:288] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:17.028735 11864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:17.029399 11864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [term 1 LEADER]: Becoming Leader. State: Replica: 0eb945ba8b2d42e0bd50b0e21b9bc761, State: Running, Role: LEADER
I20260927 14:01:17.029922 11864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb945ba8b2d42e0bd50b0e21b9bc761" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260927 14:01:17.030579 11854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:17.032932 11865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eb945ba8b2d42e0bd50b0e21b9bc761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb945ba8b2d42e0bd50b0e21b9bc761" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } } }
I20260927 14:01:17.033108 11866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eb945ba8b2d42e0bd50b0e21b9bc761. Latest consensus state: current_term: 1 leader_uuid: "0eb945ba8b2d42e0bd50b0e21b9bc761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb945ba8b2d42e0bd50b0e21b9bc761" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } } }
I20260927 14:01:17.033352 11865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:17.033370 11866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:17.035636 11871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:17.040959 11871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:17.049813 11871 catalog_manager.cc:1347] Generated new cluster ID: a2783d546e4148078dee2e3d40648840
I20260927 14:01:17.049954 11871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:17.070838 11871 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:17.071339 11871 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:17.078692 11871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eb945ba8b2d42e0bd50b0e21b9bc761: Generated new TSK 0
I20260927 14:01:17.079824 11871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:17.093545 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11787
W20260927 14:01:17.095372 11849 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.58:55949
W20260927 14:01:17.096180 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11787: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 14:01:17.096346 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 11787
2026-09-27T14:01:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:17 dist-test-slave-gkks krb5kdc[11885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:17 dist-test-slave-gkks krb5kdc[11885](info): set up 2 sockets
Sep 27 14:01:17 dist-test-slave-gkks krb5kdc[11885](info): commencing operation
krb5kdc: starting...
W20260927 14:01:18.195714 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:18 dist-test-slave-gkks krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 14:01:18.462754 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:46051
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:46051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:18.791286 11898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:18.791570 11898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:18.791777 11898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:18.799043 11898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:18.799211 11898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:18.799317 11898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:18.799428 11898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:18.799497 11898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:18.809429 11898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:46051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:46051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:18.810770 11898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:18.812108 11898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:18.823642 11903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:18.823702 11906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:18.823740 11904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:18.824679 11898 server_base.cc:1034] running on GCE node
I20260927 14:01:18.825398 11898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:18.826785 11898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:18.828001 11898 hybrid_clock.cc:648] HybridClock initialized: now 1790517678827953 us; error 33 us; skew 500 ppm
I20260927 14:01:18.828418 11898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:18 dist-test-slave-gkks krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517678, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:18.831532 11898 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 14:01:18.833163 11898 webserver.cc:460] Webserver started at http://127.22.26.254:45299/ using document root <none> and password file <none>
I20260927 14:01:18.833901 11898 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:18.834056 11898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:18.834372 11898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:18.837512 11898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "907265f0d7674a7cb62f74e4b09de547"
format_stamp: "Formatted at 2026-09-27 14:01:18 on dist-test-slave-gkks"
server_key: "9b94b4f3a6d72803c3a8661ab8b46b88"
server_key_iv: "6c6b2daf3e6f36d587ddb1e2f4ceede6"
server_key_version: "encryptionkey@0"
I20260927 14:01:18.838317 11898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "907265f0d7674a7cb62f74e4b09de547"
format_stamp: "Formatted at 2026-09-27 14:01:18 on dist-test-slave-gkks"
server_key: "9b94b4f3a6d72803c3a8661ab8b46b88"
server_key_iv: "6c6b2daf3e6f36d587ddb1e2f4ceede6"
server_key_version: "encryptionkey@0"
I20260927 14:01:18.843892 11898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260927 14:01:18.847559 11913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:18.849009 11898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 14:01:18.849216 11898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "907265f0d7674a7cb62f74e4b09de547"
format_stamp: "Formatted at 2026-09-27 14:01:18 on dist-test-slave-gkks"
server_key: "9b94b4f3a6d72803c3a8661ab8b46b88"
server_key_iv: "6c6b2daf3e6f36d587ddb1e2f4ceede6"
server_key_version: "encryptionkey@0"
I20260927 14:01:18.849447 11898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:18.869589 11898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:18.873998 11898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:18.874320 11898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:18.894522 11898 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:46051
I20260927 14:01:18.894544 11964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:46051 every 8 connection(s)
I20260927 14:01:18.896073 11898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:18.897496 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 11898
I20260927 14:01:18.897670 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:18.898231 22635 external_mini_cluster.cc:1428] Setting key b1be9ed98cfd0229e9824c30929e41a2
I20260927 14:01:18.900794 11965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 14:01:18 dist-test-slave-gkks krb5kdc[11885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:01:18.914530 11965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547: Bootstrap starting.
I20260927 14:01:18.918886 11965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:18.920182 11965 log.cc:824] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:18.923615 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:18.900147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49197 (local address 127.22.26.254:46051)
0927 14:01:18.901000 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:18.901012 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:18.901053 (+ 41us) server_negotiation.cc:407] Connection header received
0927 14:01:18.902690 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:18.902713 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:18.903182 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:18.903553 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:18.905422 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:18.906831 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:18.908719 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:18.909434 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:18.909581 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:18.914591 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:18.914603 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:18.914625 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:18.914678 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:18.918612 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:18.919637 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:18.919645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:18.919654 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:18.919870 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:18.920361 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:18.920366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:18.920369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:18.921056 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:18.921300 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:18.921915 (+ 615us) server_negotiation.cc:299] Negotiation successful
0927 14:01:18.922275 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":374,"threads_started":1}
I20260927 14:01:18.923764 11965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547: No bootstrap required, opened a new log
I20260927 14:01:18.927686 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:18.932165 11965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907265f0d7674a7cb62f74e4b09de547" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 46051 } }
I20260927 14:01:18.932628 11965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:18.932747 11965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907265f0d7674a7cb62f74e4b09de547, State: Initialized, Role: FOLLOWER
I20260927 14:01:18.933387 11965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907265f0d7674a7cb62f74e4b09de547" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 46051 } }
I20260927 14:01:18.933626 11965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:18.933780 11965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:18.933960 11965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:18.936285 11965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907265f0d7674a7cb62f74e4b09de547" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 46051 } }
I20260927 14:01:18.936784 11965 leader_election.cc:302] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 907265f0d7674a7cb62f74e4b09de547; no voters:
I20260927 14:01:18.937439 11965 leader_election.cc:288] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:18.937552 11975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:18.939126 11975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [term 1 LEADER]: Becoming Leader. State: Replica: 907265f0d7674a7cb62f74e4b09de547, State: Running, Role: LEADER
I20260927 14:01:18.939797 11975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907265f0d7674a7cb62f74e4b09de547" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 46051 } }
I20260927 14:01:18.940487 11965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:18.945554 11976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "907265f0d7674a7cb62f74e4b09de547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907265f0d7674a7cb62f74e4b09de547" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 46051 } } }
I20260927 14:01:18.945976 11976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:18.946364 11977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907265f0d7674a7cb62f74e4b09de547. Latest consensus state: current_term: 1 leader_uuid: "907265f0d7674a7cb62f74e4b09de547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907265f0d7674a7cb62f74e4b09de547" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 46051 } } }
I20260927 14:01:18.946710 11977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:18.951439 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:18.932178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39035 (local address 127.22.26.254:46051)
0927 14:01:18.932368 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:18.932374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:18.932893 (+ 519us) server_negotiation.cc:407] Connection header received
0927 14:01:18.933087 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:18.933093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:18.933171 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:18.933272 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:18.934948 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:18.935866 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:18.937248 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:18.937967 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:18.938101 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:18.941168 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:18.941174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:18.941178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:18.941214 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:18.945575 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:18.946477 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:18.946484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:18.946491 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:18.947211 (+ 720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:18.947994 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:18.948002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:18.948008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:18.948521 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:18.948924 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:18.950621 (+ 1697us) server_negotiation.cc:299] Negotiation successful
0927 14:01:18.950880 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 14:01:18.959416 11984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:18.964197 11984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:18.973166 11984 catalog_manager.cc:1347] Generated new cluster ID: 63776f8dfa924fc7acece35f63ff59bc
I20260927 14:01:18.973332 11984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:19.003095 11984 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:19.005040 11984 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:19.016201 11984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 907265f0d7674a7cb62f74e4b09de547: Generated new TSK 0
I20260927 14:01:19.017293 11984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:19.023962 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 11898
I20260927 14:01:19.025198 11960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:01:19.119383 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 11898
W20260927 14:01:19.136934 11883 connection.cc:535] client connection to 127.22.26.254:46051 recv error: Network error: failed to read from TLS socket (remote: 127.22.26.254:46051): Connection reset by peer (error 104)
2026-09-27T14:01:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2025 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:19 dist-test-slave-gkks krb5kdc[11997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:19 dist-test-slave-gkks krb5kdc[11997](info): set up 2 sockets
Sep 27 14:01:19 dist-test-slave-gkks krb5kdc[11997](info): commencing operation
krb5kdc: starting...
W20260927 14:01:20.231722 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:20 dist-test-slave-gkks krb5kdc[11997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 14:01:20.494706 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42539
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:42539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:20.826879 12010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:20.827293 12010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:20.827461 12010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:20.834656 12010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:20.834820 12010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:20.834918 12010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:20.835028 12010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:20.835141 12010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:20.844895 12010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:42539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:20.846187 12010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:20.847548 12010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:20.859382 12015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:20.859354 12018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:20.859362 12016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:20.860579 12010 server_base.cc:1034] running on GCE node
I20260927 14:01:20.861375 12010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:20.862762 12010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:20.864018 12010 hybrid_clock.cc:648] HybridClock initialized: now 1790517680863964 us; error 45 us; skew 500 ppm
I20260927 14:01:20.864400 12010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:20 dist-test-slave-gkks krb5kdc[11997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517680, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:20.867497 12010 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 14:01:20.869134 12010 webserver.cc:460] Webserver started at http://127.22.26.254:43865/ using document root <none> and password file <none>
I20260927 14:01:20.869910 12010 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:20.870067 12010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:20.870383 12010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:20.873452 12010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "e7dd9a8460b448d4a9e8b319cbb50bad"
format_stamp: "Formatted at 2026-09-27 14:01:20 on dist-test-slave-gkks"
server_key: "47ce723f810f40a029111970727ba173"
server_key_iv: "16e880bbb6466197e893de944039c842"
server_key_version: "encryptionkey@0"
I20260927 14:01:20.874218 12010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "e7dd9a8460b448d4a9e8b319cbb50bad"
format_stamp: "Formatted at 2026-09-27 14:01:20 on dist-test-slave-gkks"
server_key: "47ce723f810f40a029111970727ba173"
server_key_iv: "16e880bbb6466197e893de944039c842"
server_key_version: "encryptionkey@0"
I20260927 14:01:20.879731 12010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 14:01:20.883334 12025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:20.884760 12010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:01:20.884955 12010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "e7dd9a8460b448d4a9e8b319cbb50bad"
format_stamp: "Formatted at 2026-09-27 14:01:20 on dist-test-slave-gkks"
server_key: "47ce723f810f40a029111970727ba173"
server_key_iv: "16e880bbb6466197e893de944039c842"
server_key_version: "encryptionkey@0"
I20260927 14:01:20.885162 12010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:20.910218 12010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:20.914619 12010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:20.914932 12010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:20.933941 12010 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:42539
I20260927 14:01:20.933965 12076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:42539 every 8 connection(s)
I20260927 14:01:20.935420 12010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:20.939096 12077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:20.940232 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 12010
I20260927 14:01:20.940428 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:20.941005 22635 external_mini_cluster.cc:1428] Setting key 6de45815ab256a8a033b335a58518b59
I20260927 14:01:20.953437 12077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad: Bootstrap starting.
I20260927 14:01:20.954897 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:20.942910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41821 (local address 127.22.26.254:42539)
0927 14:01:20.943791 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:20.943804 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:20.943849 (+ 45us) server_negotiation.cc:407] Connection header received
0927 14:01:20.945489 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:20.945512 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:20.945918 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:20.946261 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:20.946870 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:20.948208 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:20.950145 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:20.950843 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:20.951336 (+ 493us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:20.952602 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:20.952610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:20.952624 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:20.953161 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:20.953235 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:20.953259 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:20.953367 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:20.953755 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 14:01:20.953949 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":441,"threads_started":1}
I20260927 14:01:20.956768 12077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:20.957983 12077 log.cc:824] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:20.958623 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:20.960714 12077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad: No bootstrap required, opened a new log
I20260927 14:01:20.969828 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:20.963122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45037 (local address 127.22.26.254:42539)
0927 14:01:20.963306 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:20.963312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:20.963800 (+ 488us) server_negotiation.cc:407] Connection header received
0927 14:01:20.963941 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:20.963951 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:20.964032 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:20.964140 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:20.964583 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:20.965708 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:20.967107 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:20.967832 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:20.967969 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:20.969075 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:20.969081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:20.969087 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:20.969107 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:20.969165 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:20.969185 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:20.969295 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:20.969458 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 14:01:20.969509 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:01:20.970127 12077 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7dd9a8460b448d4a9e8b319cbb50bad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42539 } }
I20260927 14:01:20.970577 12077 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:20.970695 12077 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7dd9a8460b448d4a9e8b319cbb50bad, State: Initialized, Role: FOLLOWER
I20260927 14:01:20.971313 12077 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7dd9a8460b448d4a9e8b319cbb50bad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42539 } }
I20260927 14:01:20.971516 12077 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:20.971611 12077 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:20.971733 12077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:20.973994 12077 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7dd9a8460b448d4a9e8b319cbb50bad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42539 } }
I20260927 14:01:20.974449 12077 leader_election.cc:302] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7dd9a8460b448d4a9e8b319cbb50bad; no voters:
I20260927 14:01:20.975194 12077 leader_election.cc:288] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:20.975355 12087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:20.976125 12087 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [term 1 LEADER]: Becoming Leader. State: Replica: e7dd9a8460b448d4a9e8b319cbb50bad, State: Running, Role: LEADER
I20260927 14:01:20.976733 12087 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7dd9a8460b448d4a9e8b319cbb50bad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42539 } }
I20260927 14:01:20.977415 12077 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:20.980191 12088 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7dd9a8460b448d4a9e8b319cbb50bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7dd9a8460b448d4a9e8b319cbb50bad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42539 } } }
I20260927 14:01:20.979863 12089 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7dd9a8460b448d4a9e8b319cbb50bad. Latest consensus state: current_term: 1 leader_uuid: "e7dd9a8460b448d4a9e8b319cbb50bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7dd9a8460b448d4a9e8b319cbb50bad" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42539 } } }
I20260927 14:01:20.980603 12089 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:20.980580 12088 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:20.981557 12094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:20.986883 12094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:20.996974 12094 catalog_manager.cc:1347] Generated new cluster ID: b70ab5c79b0a440d9a4351846944dd58
I20260927 14:01:20.997092 12094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:21.012543 12094 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:21.014102 12094 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:21.021433 12094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7dd9a8460b448d4a9e8b319cbb50bad: Generated new TSK 0
I20260927 14:01:21.022418 12094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:21.036963 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 12010
W20260927 14:01:21.038578 12072 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41821
W20260927 14:01:21.039315 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 12010: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 14:01:21.039469 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 12010
W20260927 14:01:21.056516 11995 connection.cc:535] client connection to 127.22.26.254:42539 recv error: Network error: failed to read from TLS socket (remote: 127.22.26.254:42539): Connection reset by peer (error 104)
2026-09-27T14:01:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:21 dist-test-slave-gkks krb5kdc[12108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:21 dist-test-slave-gkks krb5kdc[12108](info): set up 2 sockets
Sep 27 14:01:21 dist-test-slave-gkks krb5kdc[12108](info): commencing operation
krb5kdc: starting...
W20260927 14:01:22.139676 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:22 dist-test-slave-gkks krb5kdc[12108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 14:01:22.399524 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:34047
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:34047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:22.726965 12121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:22.727276 12121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:22.727466 12121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:22.734597 12121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:22.734699 12121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:22.734793 12121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:22.734884 12121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:22.734968 12121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:22.745265 12121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:34047
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:34047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:22.746567 12121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:22.747973 12121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:22.759826 12126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:22.759881 12127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:22.759819 12129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:22.760820 12121 server_base.cc:1034] running on GCE node
I20260927 14:01:22.761626 12121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:22.763164 12121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:22.764397 12121 hybrid_clock.cc:648] HybridClock initialized: now 1790517682764350 us; error 41 us; skew 500 ppm
I20260927 14:01:22.764776 12121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:22 dist-test-slave-gkks krb5kdc[12108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517682, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:22.773095 12121 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 14:01:22.774676 12121 webserver.cc:460] Webserver started at http://127.22.26.254:44965/ using document root <none> and password file <none>
I20260927 14:01:22.775490 12121 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:22.775596 12121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:22.775854 12121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:22.778872 12121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "6e26a3b5027344738515b4cb0672a949"
format_stamp: "Formatted at 2026-09-27 14:01:22 on dist-test-slave-gkks"
server_key: "c3ad0645df20b91c5fd6723252bd9e17"
server_key_iv: "5518da5e3a8a29f25fd905cf697b6ce0"
server_key_version: "encryptionkey@0"
I20260927 14:01:22.779714 12121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "6e26a3b5027344738515b4cb0672a949"
format_stamp: "Formatted at 2026-09-27 14:01:22 on dist-test-slave-gkks"
server_key: "c3ad0645df20b91c5fd6723252bd9e17"
server_key_iv: "5518da5e3a8a29f25fd905cf697b6ce0"
server_key_version: "encryptionkey@0"
I20260927 14:01:22.785486 12121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 14:01:22.789110 12136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:22.790482 12121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:01:22.790622 12121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "6e26a3b5027344738515b4cb0672a949"
format_stamp: "Formatted at 2026-09-27 14:01:22 on dist-test-slave-gkks"
server_key: "c3ad0645df20b91c5fd6723252bd9e17"
server_key_iv: "5518da5e3a8a29f25fd905cf697b6ce0"
server_key_version: "encryptionkey@0"
I20260927 14:01:22.790788 12121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:22.796355 12121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:22.800310 12121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:22.800550 12121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:22.819749 12121 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:34047
I20260927 14:01:22.819779 12187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:34047 every 8 connection(s)
I20260927 14:01:22.821138 12121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:22.824086 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 12121
I20260927 14:01:22.824298 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:22.824765 22635 external_mini_cluster.cc:1428] Setting key e9872c6ff50a933675fc58187897b43d
I20260927 14:01:22.824946 12188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:22.832721 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:22.826527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35855 (local address 127.22.26.254:34047)
0927 14:01:22.827345 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:22.827355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:22.827395 (+ 40us) server_negotiation.cc:407] Connection header received
0927 14:01:22.829395 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:22.829415 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:22.829745 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:22.830129 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:22.831200 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:22.831206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:22.831218 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:22.831238 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:22.831293 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:22.831314 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:22.831558 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:22.831709 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 14:01:22.831873 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":329,"threads_started":1}
I20260927 14:01:22.836203 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:22.837558 12188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949: Bootstrap starting.
I20260927 14:01:22.840808 12188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:22.841852 12188 log.cc:824] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:22.843061 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:22.840510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42895 (local address 127.22.26.254:34047)
0927 14:01:22.840684 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:22.840695 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:22.841079 (+ 384us) server_negotiation.cc:407] Connection header received
0927 14:01:22.841274 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:22.841279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:22.841332 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:22.841432 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:22.842381 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:22.842387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:22.842393 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:22.842413 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:22.842448 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:22.842462 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:22.842591 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:22.842744 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 14:01:22.842808 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:01:22.844794 12188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949: No bootstrap required, opened a new log
I20260927 14:01:22.852149 12188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e26a3b5027344738515b4cb0672a949" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 34047 } }
I20260927 14:01:22.852515 12188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:22.852620 12188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e26a3b5027344738515b4cb0672a949, State: Initialized, Role: FOLLOWER
I20260927 14:01:22.853215 12188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e26a3b5027344738515b4cb0672a949" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 34047 } }
I20260927 14:01:22.853420 12188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:22.853519 12188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:22.853638 12188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:22.856020 12188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e26a3b5027344738515b4cb0672a949" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 34047 } }
I20260927 14:01:22.856446 12188 leader_election.cc:302] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e26a3b5027344738515b4cb0672a949; no voters:
I20260927 14:01:22.856961 12188 leader_election.cc:288] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:22.857112 12198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:22.857791 12198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [term 1 LEADER]: Becoming Leader. State: Replica: 6e26a3b5027344738515b4cb0672a949, State: Running, Role: LEADER
I20260927 14:01:22.858275 12198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e26a3b5027344738515b4cb0672a949" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 34047 } }
I20260927 14:01:22.859022 12188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:22.860996 12200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e26a3b5027344738515b4cb0672a949. Latest consensus state: current_term: 1 leader_uuid: "6e26a3b5027344738515b4cb0672a949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e26a3b5027344738515b4cb0672a949" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 34047 } } }
I20260927 14:01:22.861603 12200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:22.861825 12199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e26a3b5027344738515b4cb0672a949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e26a3b5027344738515b4cb0672a949" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 34047 } } }
I20260927 14:01:22.862193 12199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:22.863981 12203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:22.870749 12203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:22.881392 12203 catalog_manager.cc:1347] Generated new cluster ID: d9c54cf7ba4c408aadf3e7d359b0d10f
I20260927 14:01:22.881510 12203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:22.895378 12203 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:22.895866 12203 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:22.906266 12203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e26a3b5027344738515b4cb0672a949: Generated new TSK 0
I20260927 14:01:22.907244 12203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:22.920678 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 12121
W20260927 14:01:22.922165 12183 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35855
W20260927 14:01:22.923033 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 12121: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 14:01:22.923224 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 12121
2026-09-27T14:01:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:23 dist-test-slave-gkks krb5kdc[12219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:23 dist-test-slave-gkks krb5kdc[12219](info): set up 2 sockets
Sep 27 14:01:23 dist-test-slave-gkks krb5kdc[12219](info): commencing operation
krb5kdc: starting...
W20260927 14:01:24.035759 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:24 dist-test-slave-gkks krb5kdc[12219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 14:01:24.299911 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:37923
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:37923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:24.634420 12232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:24.634697 12232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:24.634886 12232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:24.642071 12232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:24.642200 12232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:24.642292 12232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:24.642374 12232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:24.642442 12232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:24.652380 12232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:37923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:37923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:24.653651 12232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:24.654958 12232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:24.666890 12240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:24.666846 12237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:24.666929 12238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:24.667822 12232 server_base.cc:1034] running on GCE node
I20260927 14:01:24.668614 12232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:24.669961 12232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:24.671191 12232 hybrid_clock.cc:648] HybridClock initialized: now 1790517684671126 us; error 63 us; skew 500 ppm
I20260927 14:01:24.671540 12232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:24 dist-test-slave-gkks krb5kdc[12219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517684, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:24.674820 12232 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 14:01:24.676309 12232 webserver.cc:460] Webserver started at http://127.22.26.254:36207/ using document root <none> and password file <none>
I20260927 14:01:24.676954 12232 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:24.677049 12232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:24.677297 12232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:24.680244 12232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "5e6432c8033e49f683ccad38c290a9aa"
format_stamp: "Formatted at 2026-09-27 14:01:24 on dist-test-slave-gkks"
server_key: "8b9e4bb4a1804937ef7940470412b847"
server_key_iv: "96c6ba69732ac53ac92b3fe49dd7bb09"
server_key_version: "encryptionkey@0"
I20260927 14:01:24.680979 12232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "5e6432c8033e49f683ccad38c290a9aa"
format_stamp: "Formatted at 2026-09-27 14:01:24 on dist-test-slave-gkks"
server_key: "8b9e4bb4a1804937ef7940470412b847"
server_key_iv: "96c6ba69732ac53ac92b3fe49dd7bb09"
server_key_version: "encryptionkey@0"
I20260927 14:01:24.686049 12232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 14:01:24.689393 12247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:24.690661 12232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 14:01:24.690857 12232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "5e6432c8033e49f683ccad38c290a9aa"
format_stamp: "Formatted at 2026-09-27 14:01:24 on dist-test-slave-gkks"
server_key: "8b9e4bb4a1804937ef7940470412b847"
server_key_iv: "96c6ba69732ac53ac92b3fe49dd7bb09"
server_key_version: "encryptionkey@0"
I20260927 14:01:24.691039 12232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:24.709695 12232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:24.713968 12232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:24.714268 12232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:24.733316 12232 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:37923
I20260927 14:01:24.733340 12298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:37923 every 8 connection(s)
I20260927 14:01:24.734750 12232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:24.735330 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 12232
I20260927 14:01:24.735554 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:24.736141 22635 external_mini_cluster.cc:1428] Setting key a1b4619e8baa631dc5536a6d2e38926d
I20260927 14:01:24.739537 12299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 14:01:24 dist-test-slave-gkks krb5kdc[12219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:01:24.750492 12299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa: Bootstrap starting.
I20260927 14:01:24.754773 12299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:24.756150 12299 log.cc:824] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:24.759197 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:24.737991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33737 (local address 127.22.26.254:37923)
0927 14:01:24.738862 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:24.738874 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:24.738916 (+ 42us) server_negotiation.cc:407] Connection header received
0927 14:01:24.740595 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:24.740617 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:24.740986 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:24.741358 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:24.742821 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:24.744235 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:24.745530 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:24.746250 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:24.746450 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:24.750538 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:24.750549 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:24.750569 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:24.750616 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:24.754484 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:24.755285 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:24.755294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:24.755302 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:24.755515 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:24.755886 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:24.755896 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:24.755901 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:24.756692 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:24.757001 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:24.757451 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 14:01:24.757881 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":423,"threads_started":1}
I20260927 14:01:24.759672 12299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa: No bootstrap required, opened a new log
I20260927 14:01:24.762827 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:24.768029 12299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6432c8033e49f683ccad38c290a9aa" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37923 } }
I20260927 14:01:24.768484 12299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:24.768582 12299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e6432c8033e49f683ccad38c290a9aa, State: Initialized, Role: FOLLOWER
I20260927 14:01:24.769222 12299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6432c8033e49f683ccad38c290a9aa" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37923 } }
I20260927 14:01:24.769446 12299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:24.769578 12299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:24.769775 12299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:24.772598 12299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6432c8033e49f683ccad38c290a9aa" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37923 } }
I20260927 14:01:24.773213 12299 leader_election.cc:302] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e6432c8033e49f683ccad38c290a9aa; no voters:
I20260927 14:01:24.773876 12299 leader_election.cc:288] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:24.774024 12309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:24.775833 12299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:24.776482 12309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [term 1 LEADER]: Becoming Leader. State: Replica: 5e6432c8033e49f683ccad38c290a9aa, State: Running, Role: LEADER
I20260927 14:01:24.777070 12309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6432c8033e49f683ccad38c290a9aa" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37923 } }
I20260927 14:01:24.780263 12311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e6432c8033e49f683ccad38c290a9aa. Latest consensus state: current_term: 1 leader_uuid: "5e6432c8033e49f683ccad38c290a9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6432c8033e49f683ccad38c290a9aa" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37923 } } }
I20260927 14:01:24.780253 12310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e6432c8033e49f683ccad38c290a9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6432c8033e49f683ccad38c290a9aa" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37923 } } }
I20260927 14:01:24.780692 12311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:24.780745 12310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:24.785874 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:24.767152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47513 (local address 127.22.26.254:37923)
0927 14:01:24.767336 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:24.767343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:24.767788 (+ 445us) server_negotiation.cc:407] Connection header received
0927 14:01:24.767920 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:24.767925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:24.768002 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:24.768116 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:24.769702 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:24.770863 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:24.772126 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:24.772847 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:24.772978 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:24.776224 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:24.776230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:24.776239 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:24.776278 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:24.781032 (+ 4754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:24.782126 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:24.782134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:24.782140 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:24.782554 (+ 414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:24.783110 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:24.783119 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:24.783124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:24.783590 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:24.783978 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:24.785068 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0927 14:01:24.785376 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 14:01:24.788280 12315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:24.793393 12315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:24.802944 12315 catalog_manager.cc:1347] Generated new cluster ID: 9b885c46f1944e0084f09dede42dd0d4
I20260927 14:01:24.803143 12315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:24.830960 12315 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:24.832345 12315 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:24.844236 12315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e6432c8033e49f683ccad38c290a9aa: Generated new TSK 0
I20260927 14:01:24.845208 12315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:24.852520 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 12232
I20260927 14:01:24.853690 12294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:01:24.946834 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 12232
2026-09-27T14:01:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:25 dist-test-slave-gkks krb5kdc[12331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:25 dist-test-slave-gkks krb5kdc[12331](info): set up 2 sockets
Sep 27 14:01:25 dist-test-slave-gkks krb5kdc[12331](info): commencing operation
krb5kdc: starting...
W20260927 14:01:26.059899 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:26 dist-test-slave-gkks krb5kdc[12331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 14:01:26.322187 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:33287
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:33287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:26.654712 12344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:26.655059 12344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:26.655313 12344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:26.662791 12344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:26.662916 12344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:26.662999 12344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:26.663149 12344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:26.663218 12344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:26.673583 12344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:33287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:33287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:26.675027 12344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:26.676569 12344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:26.689162 12349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:26.689214 12350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:26.689188 12352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:26.690079 12344 server_base.cc:1034] running on GCE node
I20260927 14:01:26.690824 12344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:26.692320 12344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:26.693569 12344 hybrid_clock.cc:648] HybridClock initialized: now 1790517686693499 us; error 62 us; skew 500 ppm
I20260927 14:01:26.693962 12344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:26 dist-test-slave-gkks krb5kdc[12331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517686, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:26.697276 12344 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 14:01:26.698712 12344 webserver.cc:460] Webserver started at http://127.22.26.254:35005/ using document root <none> and password file <none>
I20260927 14:01:26.699445 12344 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:26.699548 12344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:26.699810 12344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:26.702704 12344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "27677494ac32453a8ab79b352d485cec"
format_stamp: "Formatted at 2026-09-27 14:01:26 on dist-test-slave-gkks"
server_key: "507e62683645eb9ae190d4b3c3663c2f"
server_key_iv: "c117387a3fa6095d08e244c3f4183221"
server_key_version: "encryptionkey@0"
I20260927 14:01:26.703501 12344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "27677494ac32453a8ab79b352d485cec"
format_stamp: "Formatted at 2026-09-27 14:01:26 on dist-test-slave-gkks"
server_key: "507e62683645eb9ae190d4b3c3663c2f"
server_key_iv: "c117387a3fa6095d08e244c3f4183221"
server_key_version: "encryptionkey@0"
I20260927 14:01:26.708568 12344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 14:01:26.712013 12359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:26.713379 12344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 14:01:26.713667 12344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "27677494ac32453a8ab79b352d485cec"
format_stamp: "Formatted at 2026-09-27 14:01:26 on dist-test-slave-gkks"
server_key: "507e62683645eb9ae190d4b3c3663c2f"
server_key_iv: "c117387a3fa6095d08e244c3f4183221"
server_key_version: "encryptionkey@0"
I20260927 14:01:26.713871 12344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:26.737550 12344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:26.741928 12344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:26.742233 12344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:26.761631 12344 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:33287
I20260927 14:01:26.761646 12410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:33287 every 8 connection(s)
I20260927 14:01:26.763063 12344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:26.766840 12411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:26.767493 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 12344
I20260927 14:01:26.767685 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:26.768157 22635 external_mini_cluster.cc:1428] Setting key 7a5448421c6fc1b0cbbafe99e94c1605
I20260927 14:01:26.780545 12411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec: Bootstrap starting.
I20260927 14:01:26.782037 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:26.769935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41569 (local address 127.22.26.254:33287)
0927 14:01:26.770747 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:26.770759 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:26.770799 (+ 40us) server_negotiation.cc:407] Connection header received
0927 14:01:26.772874 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:26.772900 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:26.773312 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:26.773651 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:26.774248 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:26.775599 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:26.777565 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:26.778271 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:26.778410 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:26.779768 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:26.779776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:26.779789 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:26.780327 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:26.780401 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:26.780425 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:26.780510 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:26.780793 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 14:01:26.781088 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":384,"threads_started":1}
I20260927 14:01:26.783846 12411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:26.785081 12411 log.cc:824] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:26.785980 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:26.788478 12411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec: No bootstrap required, opened a new log
I20260927 14:01:26.798794 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:26.791727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59061 (local address 127.22.26.254:33287)
0927 14:01:26.791928 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:26.791936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:26.792395 (+ 459us) server_negotiation.cc:407] Connection header received
0927 14:01:26.792572 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:26.792578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:26.792653 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:26.792777 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:26.793336 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:26.794483 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:26.795963 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:26.796740 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:26.796883 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:26.798028 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:26.798034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:26.798039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:26.798087 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:26.798136 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:26.798157 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:26.798262 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:26.798402 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 14:01:26.798500 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 14:01:26.798452 12411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27677494ac32453a8ab79b352d485cec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33287 } }
I20260927 14:01:26.798969 12411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:26.799109 12411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27677494ac32453a8ab79b352d485cec, State: Initialized, Role: FOLLOWER
I20260927 14:01:26.799783 12411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27677494ac32453a8ab79b352d485cec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33287 } }
I20260927 14:01:26.800035 12411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:26.800185 12411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:26.800364 12411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:26.802556 12411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27677494ac32453a8ab79b352d485cec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33287 } }
I20260927 14:01:26.803064 12411 leader_election.cc:302] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27677494ac32453a8ab79b352d485cec; no voters:
I20260927 14:01:26.803709 12411 leader_election.cc:288] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:26.803876 12421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:26.804687 12421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [term 1 LEADER]: Becoming Leader. State: Replica: 27677494ac32453a8ab79b352d485cec, State: Running, Role: LEADER
I20260927 14:01:26.805223 12421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27677494ac32453a8ab79b352d485cec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33287 } }
I20260927 14:01:26.805915 12411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:26.808755 12422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27677494ac32453a8ab79b352d485cec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27677494ac32453a8ab79b352d485cec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33287 } } }
I20260927 14:01:26.809216 12422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:26.809919 12423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27677494ac32453a8ab79b352d485cec. Latest consensus state: current_term: 1 leader_uuid: "27677494ac32453a8ab79b352d485cec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27677494ac32453a8ab79b352d485cec" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 33287 } } }
I20260927 14:01:26.810381 12423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:26.810581 12428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:26.824745 12428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:26.834182 12428 catalog_manager.cc:1347] Generated new cluster ID: 8f85b3b82451446c83416ce2b3ec94f9
I20260927 14:01:26.834316 12428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:26.858873 12428 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:26.860812 12428 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:26.871879 12428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27677494ac32453a8ab79b352d485cec: Generated new TSK 0
I20260927 14:01:26.872951 12428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:26.880841 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 12344
W20260927 14:01:26.882115 12406 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41569
W20260927 14:01:26.882982 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 12344: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 14:01:26.883168 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 12344
2026-09-27T14:01:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1935 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:26 dist-test-slave-gkks krb5kdc[12443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:26 dist-test-slave-gkks krb5kdc[12443](info): set up 2 sockets
Sep 27 14:01:26 dist-test-slave-gkks krb5kdc[12443](info): commencing operation
krb5kdc: starting...
W20260927 14:01:28.059926 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:28 dist-test-slave-gkks krb5kdc[12443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 14:01:28.323449 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42597
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:42597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:28.669873 12456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:28.670231 12456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:28.670403 12456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:28.677719 12456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:28.677873 12456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:28.677983 12456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:28.678100 12456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:28.678179 12456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:28.688212 12456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:42597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:42597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:28.689581 12456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:28.690977 12456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:28.703367 12462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:28.703403 12464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:28.703497 12461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:28.704080 12456 server_base.cc:1034] running on GCE node
I20260927 14:01:28.704885 12456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:28.706279 12456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:28.707510 12456 hybrid_clock.cc:648] HybridClock initialized: now 1790517688707472 us; error 33 us; skew 500 ppm
I20260927 14:01:28.707887 12456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:28 dist-test-slave-gkks krb5kdc[12443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517688, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:28.716164 12456 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 14:01:28.717763 12456 webserver.cc:460] Webserver started at http://127.22.26.254:38699/ using document root <none> and password file <none>
I20260927 14:01:28.718454 12456 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:28.718611 12456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:28.718925 12456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:28.722110 12456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "5b553a24571445979d159a9633bff9ab"
format_stamp: "Formatted at 2026-09-27 14:01:28 on dist-test-slave-gkks"
server_key: "4b6599819fa61a70aa656f72bcac4007"
server_key_iv: "f82a4b3dc60baa9d867361fc98409857"
server_key_version: "encryptionkey@0"
I20260927 14:01:28.722880 12456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "5b553a24571445979d159a9633bff9ab"
format_stamp: "Formatted at 2026-09-27 14:01:28 on dist-test-slave-gkks"
server_key: "4b6599819fa61a70aa656f72bcac4007"
server_key_iv: "f82a4b3dc60baa9d867361fc98409857"
server_key_version: "encryptionkey@0"
I20260927 14:01:28.728193 12456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 14:01:28.732108 12471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:28.733589 12456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:01:28.733803 12456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "5b553a24571445979d159a9633bff9ab"
format_stamp: "Formatted at 2026-09-27 14:01:28 on dist-test-slave-gkks"
server_key: "4b6599819fa61a70aa656f72bcac4007"
server_key_iv: "f82a4b3dc60baa9d867361fc98409857"
server_key_version: "encryptionkey@0"
I20260927 14:01:28.734016 12456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:28.739753 12456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:28.743897 12456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:28.744205 12456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:28.763054 12456 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:42597
I20260927 14:01:28.763104 12522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:42597 every 8 connection(s)
I20260927 14:01:28.764554 12456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:28.768509 12523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:28.769526 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 12456
I20260927 14:01:28.769721 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:28.770274 22635 external_mini_cluster.cc:1428] Setting key 614fb3abb58c305a804f455896866a2d
I20260927 14:01:28.778510 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:28.772217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40417 (local address 127.22.26.254:42597)
0927 14:01:28.773076 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:28.773087 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:28.773128 (+ 41us) server_negotiation.cc:407] Connection header received
0927 14:01:28.774882 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:28.774909 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:28.775333 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:28.775672 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:28.776876 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:28.776886 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:28.776900 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:28.776915 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:28.776976 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:28.776998 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:28.777161 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:28.777305 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 14:01:28.777506 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":377,"threads_started":1}
I20260927 14:01:28.782413 12523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab: Bootstrap starting.
I20260927 14:01:28.782653 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:28.785619 12523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:28.786654 12523 log.cc:824] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:28.789623 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:28.786962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59529 (local address 127.22.26.254:42597)
0927 14:01:28.787175 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:28.787184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:28.787658 (+ 474us) server_negotiation.cc:407] Connection header received
0927 14:01:28.787810 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:28.787817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:28.787896 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:28.788009 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:28.789000 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:28.789005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:28.789010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:01:28.789026 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:28.789075 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:01:28.789096 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:28.789215 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:28.789296 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 14:01:28.789353 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:01:28.790885 12523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab: No bootstrap required, opened a new log
I20260927 14:01:28.798599 12523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b553a24571445979d159a9633bff9ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42597 } }
I20260927 14:01:28.798990 12523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:01:28.799125 12523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b553a24571445979d159a9633bff9ab, State: Initialized, Role: FOLLOWER
I20260927 14:01:28.799785 12523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b553a24571445979d159a9633bff9ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42597 } }
I20260927 14:01:28.800027 12523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:01:28.800180 12523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:01:28.800359 12523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:28.802707 12523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b553a24571445979d159a9633bff9ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42597 } }
I20260927 14:01:28.803270 12523 leader_election.cc:302] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b553a24571445979d159a9633bff9ab; no voters:
I20260927 14:01:28.803894 12523 leader_election.cc:288] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:01:28.804033 12533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:28.804697 12533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [term 1 LEADER]: Becoming Leader. State: Replica: 5b553a24571445979d159a9633bff9ab, State: Running, Role: LEADER
I20260927 14:01:28.805284 12533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b553a24571445979d159a9633bff9ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42597 } }
I20260927 14:01:28.806020 12523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:28.808739 12534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b553a24571445979d159a9633bff9ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b553a24571445979d159a9633bff9ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42597 } } }
I20260927 14:01:28.809141 12534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:28.809098 12535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b553a24571445979d159a9633bff9ab. Latest consensus state: current_term: 1 leader_uuid: "5b553a24571445979d159a9633bff9ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b553a24571445979d159a9633bff9ab" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 42597 } } }
I20260927 14:01:28.809495 12535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:28.811269 12541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:28.816668 12541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:28.825709 12541 catalog_manager.cc:1347] Generated new cluster ID: 54ed677e2de74e79853cc3f48a0f5c89
I20260927 14:01:28.825839 12541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:28.854485 12541 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:28.854985 12541 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:28.866863 12541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b553a24571445979d159a9633bff9ab: Generated new TSK 0
I20260927 14:01:28.867983 12541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:28.881299 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 12456
W20260927 14:01:28.882704 12518 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40417
W20260927 14:01:28.883453 22635 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 12456: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 14:01:28.883608 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 12456
2026-09-27T14:01:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1996 ms)
[----------] 18 tests from AuthTokenIssuingTest (35176 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:28 dist-test-slave-gkks krb5kdc[12554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:28 dist-test-slave-gkks krb5kdc[12554](info): set up 2 sockets
Sep 27 14:01:28 dist-test-slave-gkks krb5kdc[12554](info): commencing operation
krb5kdc: starting...
W20260927 14:01:29.996085 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:30 dist-test-slave-gkks krb5kdc[12554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 14:01:30.262375 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:43407
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:43407,127.22.26.253:33023,127.22.26.252:33505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:30.597008 12567 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 14:01:30.597437 12567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:30.597553 12567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:30.597726 12567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:30.605273 12567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:30.605370 12567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:30.605476 12567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:30.605592 12567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:30.605722 12567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:30.615986 12567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:43407,127.22.26.253:33023,127.22.26.252:33505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:43407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:30.617393 12567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:30.618854 12567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:30.631209 12572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:30.631301 12575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:30.631264 12573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:30.632411 12567 server_base.cc:1034] running on GCE node
I20260927 14:01:30.633106 12567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:30.634424 12567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:30.635633 12567 hybrid_clock.cc:648] HybridClock initialized: now 1790517690635575 us; error 30 us; skew 500 ppm
I20260927 14:01:30.635985 12567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:30 dist-test-slave-gkks krb5kdc[12554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517690, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:30.639057 12567 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 14:01:30.640549 12567 webserver.cc:460] Webserver started at http://127.22.26.254:40971/ using document root <none> and password file <none>
I20260927 14:01:30.641242 12567 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:30.641360 12567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:30.641700 12567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:30.644732 12567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "d359099e82124452bc49adb2ad0f4e1c"
format_stamp: "Formatted at 2026-09-27 14:01:30 on dist-test-slave-gkks"
server_key: "90834acc1fd681db06ff267e9f53bf7f"
server_key_iv: "3bc966e8a800c41d7884bc2dab1b12c8"
server_key_version: "encryptionkey@0"
I20260927 14:01:30.645519 12567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "d359099e82124452bc49adb2ad0f4e1c"
format_stamp: "Formatted at 2026-09-27 14:01:30 on dist-test-slave-gkks"
server_key: "90834acc1fd681db06ff267e9f53bf7f"
server_key_iv: "3bc966e8a800c41d7884bc2dab1b12c8"
server_key_version: "encryptionkey@0"
I20260927 14:01:30.650803 12567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 14:01:30.654215 12582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:30.655575 12567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 14:01:30.655755 12567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "d359099e82124452bc49adb2ad0f4e1c"
format_stamp: "Formatted at 2026-09-27 14:01:30 on dist-test-slave-gkks"
server_key: "90834acc1fd681db06ff267e9f53bf7f"
server_key_iv: "3bc966e8a800c41d7884bc2dab1b12c8"
server_key_version: "encryptionkey@0"
I20260927 14:01:30.655970 12567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:30.675222 12567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:30.679260 12567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:30.679543 12567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:30.698242 12567 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:43407
I20260927 14:01:30.698266 12633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:43407 every 8 connection(s)
I20260927 14:01:30.699677 12567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:30.703449 12634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:30.707468 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 12567
I20260927 14:01:30.707697 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:30.708549 22635 external_mini_cluster.cc:1428] Setting key baa960e635fcabf12cd50c54b5799555
I20260927 14:01:30.711521 12634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:01:30 dist-test-slave-gkks krb5kdc[12554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517690, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.26.253@KRBTEST.COM; defaulting to no policy
I20260927 14:01:30.735529 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:30.715391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.22.26.254:43407)
0927 14:01:30.717241 (+ 1850us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:30.717247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:30.717298 (+ 51us) server_negotiation.cc:407] Connection header received
0927 14:01:30.719613 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:30.719618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:30.719845 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:30.720056 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:30.721368 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:30.722556 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:30.724000 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:30.724539 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:30.724744 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:30.728231 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:30.728239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:30.728250 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:30.728281 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:30.731208 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:30.731885 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:30.731891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:30.731894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:30.732031 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:30.732426 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:30.732437 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:30.732440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:30.732915 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:30.733119 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:30.733649 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 14:01:30.734050 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":1668,"thread_start_us":1447,"threads_started":1}
I20260927 14:01:30.735548 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:30.715391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43407 (local address 127.0.0.1:48582)
0927 14:01:30.716302 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:30.716354 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:30.719198 (+ 2844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:30.720063 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:30.720069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:30.720806 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:30.721143 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:30.721160 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:30.722718 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:30.722727 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:30.723855 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:30.723866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:30.724779 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:30.724786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:30.724980 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:30.725641 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:30.725663 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:30.728014 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:30.731339 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:30.731349 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:30.731360 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:30.731733 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:30.732142 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:30.732148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:30.732151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:30.732305 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:30.733065 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:30.733072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:30.733270 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 14:01:30.733513 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 14:01:30.734173 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":203,"thread_start_us":374,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.253@KRBTEST.COM" created.
I20260927 14:01:30.738577 12634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:01:30.739787 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:30.739387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:33023 (local address 127.0.0.1:45708)
0927 14:01:30.739556 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:30.739664 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 14:01:30.740072 12584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
I20260927 14:01:30.741546 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:30.741220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:33023 (local address 127.0.0.1:45718)
0927 14:01:30.741359 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:30.741428 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 14:01:30.741739 12634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:33023: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.253.keytab.
Entry for principal kudu/127.22.26.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.253.keytab.
I20260927 14:01:30.766423 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.253:33023
--webserver_interface=127.22.26.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.253.keytab
--principal=kudu/127.22.26.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:43407,127.22.26.253:33023,127.22.26.252:33505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:01:30.791066 12634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } attempt: 1
I20260927 14:01:30.792361 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:30.791869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:33023 (local address 127.0.0.1:45724)
0927 14:01:30.792086 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:30.792213 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 14:01:30.793358 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:30.793014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:33023 (local address 127.0.0.1:45738)
0927 14:01:30.793159 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:30.793234 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 14:01:30.793547 12634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:33023: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
I20260927 14:01:30.861811 12634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } attempt: 2
I20260927 14:01:30.863211 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:30.862603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:33023 (local address 127.0.0.1:45750)
0927 14:01:30.862883 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:30.862983 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 14:01:30.864217 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:30.863858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:33023 (local address 127.0.0.1:45754)
0927 14:01:30.864002 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:30.864093 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 14:01:30.864396 12634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:33023: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
I20260927 14:01:30.955715 12634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } attempt: 3
I20260927 14:01:30.956982 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:30.956500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:33023 (local address 127.0.0.1:45768)
0927 14:01:30.956733 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:30.956839 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 14:01:30.958114 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:30.957726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:33023 (local address 127.0.0.1:45780)
0927 14:01:30.957881 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:30.957961 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 14:01:30.958307 12634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:33023: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
I20260927 14:01:31.101580 12634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } attempt: 4
I20260927 14:01:31.102988 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.102323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:33023 (local address 127.0.0.1:45796)
0927 14:01:31.102681 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.102796 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260927 14:01:31.104209 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.103746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:33023 (local address 127.0.0.1:45812)
0927 14:01:31.103919 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.104029 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 14:01:31.104449 12634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:33023: Network error: Client connection negotiation failed: client connection to 127.22.26.253:33023: connect: Connection refused (error 111)
W20260927 14:01:31.113320 12641 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 14:01:31.113651 12641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:31.113730 12641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:31.113819 12641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:31.121937 12641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:31.122042 12641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:31.122113 12641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:31.122170 12641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:31.122247 12641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:31.133092 12641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:43407,127.22.26.253:33023,127.22.26.252:33505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.253.keytab
--principal=kudu/127.22.26.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.253:33023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.26.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:31.134366 12641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:31.135691 12641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:31.147051 12647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:31.146984 12646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:31.147100 12649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:31.148193 12641 server_base.cc:1034] running on GCE node
I20260927 14:01:31.148926 12641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:31.150260 12641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:31.151531 12641 hybrid_clock.cc:648] HybridClock initialized: now 1790517691151450 us; error 65 us; skew 500 ppm
I20260927 14:01:31.151876 12641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:31 dist-test-slave-gkks krb5kdc[12554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517691, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:31.155174 12641 init.cc:377] Logged in from keytab as kudu/127.22.26.253@KRBTEST.COM (short username kudu)
I20260927 14:01:31.156629 12641 webserver.cc:460] Webserver started at http://127.22.26.253:43863/ using document root <none> and password file <none>
I20260927 14:01:31.157322 12641 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:31.157452 12641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:31.157773 12641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:31.160888 12641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/data/instance:
uuid: "73169e62ad3a4e75b4894ea8b777e253"
format_stamp: "Formatted at 2026-09-27 14:01:31 on dist-test-slave-gkks"
server_key: "93b888ba71ca811d658e74172bcfbb7f"
server_key_iv: "aa597c3c36a7018ad64f88a00ad78d95"
server_key_version: "encryptionkey@0"
I20260927 14:01:31.161713 12641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/wal/instance:
uuid: "73169e62ad3a4e75b4894ea8b777e253"
format_stamp: "Formatted at 2026-09-27 14:01:31 on dist-test-slave-gkks"
server_key: "93b888ba71ca811d658e74172bcfbb7f"
server_key_iv: "aa597c3c36a7018ad64f88a00ad78d95"
server_key_version: "encryptionkey@0"
I20260927 14:01:31.166752 12641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 14:01:31.170171 12656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:31.171757 12641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 14:01:31.171945 12641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/wal
uuid: "73169e62ad3a4e75b4894ea8b777e253"
format_stamp: "Formatted at 2026-09-27 14:01:31 on dist-test-slave-gkks"
server_key: "93b888ba71ca811d658e74172bcfbb7f"
server_key_iv: "aa597c3c36a7018ad64f88a00ad78d95"
server_key_version: "encryptionkey@0"
I20260927 14:01:31.172185 12641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:31.189775 12641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:31.194132 12641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:31.194437 12641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:31.214411 12641 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.253:33023
I20260927 14:01:31.214442 12707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.253:33023 every 8 connection(s)
I20260927 14:01:31.215903 12641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/data/info.pb
I20260927 14:01:31.219674 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:31.222733 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 12641
I20260927 14:01:31.222924 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-1/wal/instance
I20260927 14:01:31.223489 22635 external_mini_cluster.cc:1428] Setting key b992a2905be0ab374fa45e3d01e59155
I20260927 14:01:31.228288 12708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:01:31 dist-test-slave-gkks krb5kdc[12554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517691, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.253@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.26.252@KRBTEST.COM; defaulting to no policy
I20260927 14:01:31.249282 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.231559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.22.26.254:43407)
0927 14:01:31.231781 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.231787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.233031 (+ 1244us) server_negotiation.cc:407] Connection header received
0927 14:01:31.235372 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.235377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.235469 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.235583 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.237092 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.237776 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.239349 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.239768 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.239915 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.243438 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:31.243443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:31.243447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:31.243481 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:31.246246 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.246977 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.246982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.246985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.247148 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.247574 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.247579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.247583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.247865 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:31.248019 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:31.248735 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 14:01:31.248960 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 14:01:31.250231 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.232036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43407 (local address 127.0.0.1:48590)
0927 14:01:31.232826 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.232884 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:31.235029 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:31.235820 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:31.235829 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:31.236345 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:31.236899 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.236916 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.237946 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.237950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.239196 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.239209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.239999 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.240006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.240246 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.240942 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:31.240965 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:31.243215 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:31.246401 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.246409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.246425 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.246840 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.247266 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.247271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.247275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.247454 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.247995 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:31.248002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:31.248356 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 14:01:31.248615 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 14:01:31.249106 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":297,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.252@KRBTEST.COM" created.
I20260927 14:01:31.252046 12708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:01:31 dist-test-slave-gkks krb5kdc[12554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517691, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.253@KRBTEST.COM for kudu/127.22.26.253@KRBTEST.COM
I20260927 14:01:31.267611 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.252889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:33023 (local address 127.0.0.1:45822)
0927 14:01:31.253088 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.253110 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:31.253268 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:31.254127 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:31.254130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:31.254467 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:31.254590 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.254597 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.256538 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.256542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.257354 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.257364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.258555 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.258559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.258755 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.259356 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:31.259382 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:31.262051 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:31.265143 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.265149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.265153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.265518 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.265899 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.265904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.265908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.266049 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.266614 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:31.266618 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:31.266751 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 14:01:31.266907 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 14:01:31.267192 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":40}
I20260927 14:01:31.267715 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.253005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45822 (local address 127.22.26.253:33023)
0927 14:01:31.253616 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.253626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.253669 (+ 43us) server_negotiation.cc:407] Connection header received
0927 14:01:31.253725 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.253736 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.253955 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.254117 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.254767 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.256390 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.257488 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.258408 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.258548 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.262258 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:31.262269 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:31.262282 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:31.262308 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:31.265016 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.265642 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.265656 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.265666 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.265794 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.266146 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.266153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.266157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.266492 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:31.266703 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:31.267036 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 14:01:31.267328 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":420,"thread_start_us":275,"threads_started":1}
I20260927 14:01:31.270293 12708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:01:31.271529 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.271105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:53492)
0927 14:01:31.271288 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.271382 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 14:01:31.271816 12660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
I20260927 14:01:31.273373 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.273022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:53502)
0927 14:01:31.273185 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.273269 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 14:01:31.273561 12708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33505: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.252.keytab.
Entry for principal kudu/127.22.26.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.252.keytab.
I20260927 14:01:31.287410 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.252:33505
--webserver_interface=127.22.26.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.252.keytab
--principal=kudu/127.22.26.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:43407,127.22.26.253:33023,127.22.26.252:33505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:01:31.322861 12708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } attempt: 1
I20260927 14:01:31.324128 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.323657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:53504)
0927 14:01:31.323864 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.323979 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 14:01:31.325110 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.324775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:53520)
0927 14:01:31.324945 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.325011 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 14:01:31.325318 12708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33505: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
I20260927 14:01:31.393613 12708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } attempt: 2
I20260927 14:01:31.394888 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.394383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:53526)
0927 14:01:31.394614 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.394723 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 14:01:31.396032 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.395681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:53534)
0927 14:01:31.395826 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.395907 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 14:01:31.396268 12708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33505: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
I20260927 14:01:31.403775 12634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } attempt: 5
Sep 27 14:01:31 dist-test-slave-gkks krb5kdc[12554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517690, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for kudu/127.22.26.253@KRBTEST.COM
I20260927 14:01:31.416853 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.404442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:33023 (local address 127.0.0.1:45828)
0927 14:01:31.404643 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.404663 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:31.404806 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:31.405159 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:31.405164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:31.405572 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:31.405753 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.405762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.406729 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.406737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.407666 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.407678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.408431 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.408437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.408675 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.409379 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:31.409399 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:31.411470 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:31.414427 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.414432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.414437 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.414803 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.415185 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.415190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.415194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.415344 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.415864 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:31.415870 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:31.416037 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 14:01:31.416185 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 14:01:31.416483 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 14:01:31.416853 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.404517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.22.26.253:33023)
0927 14:01:31.404695 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.404733 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.404757 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:01:31.404947 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.404954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.405022 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.405125 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.405899 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.406596 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.407816 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.408265 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.408424 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.411658 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:31.411663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:31.411666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:31.411697 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:31.414283 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.414918 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.414925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.414929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.415050 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.415453 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.415458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.415461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.415744 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:31.415901 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:31.416318 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 14:01:31.416531 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 14:01:31.418118 12634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:01:31.419261 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.418838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54480)
0927 14:01:31.419003 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.419121 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 14:01:31.420444 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.420022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54494)
0927 14:01:31.420208 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.420310 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 14:01:31.420713 12634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33505: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
I20260927 14:01:31.471987 12634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } attempt: 1
I20260927 14:01:31.473404 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.472747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54506)
0927 14:01:31.473016 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.473152 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 14:01:31.474447 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.474097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54518)
0927 14:01:31.474245 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.474324 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 14:01:31.474700 12634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33505: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
I20260927 14:01:31.487519 12708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } attempt: 3
I20260927 14:01:31.488560 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.488115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54534)
0927 14:01:31.488305 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.488414 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 14:01:31.489588 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.489183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54540)
0927 14:01:31.489405 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.489485 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260927 14:01:31.489849 12708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33505: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
I20260927 14:01:31.542960 12634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } attempt: 2
I20260927 14:01:31.544270 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.543690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54544)
0927 14:01:31.543939 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.544058 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 14:01:31.545361 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.544967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54552)
0927 14:01:31.545120 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.545211 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 14:01:31.545584 12634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33505: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
W20260927 14:01:31.621886 12715 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 14:01:31.622203 12715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:31.622268 12715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:31.622339 12715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:31.629550 12715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:31.629637 12715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:31.629691 12715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:31.629742 12715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:31.629799 12715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:31.633068 12708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } attempt: 4
I20260927 14:01:31.634297 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.633770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54560)
0927 14:01:31.634018 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.634112 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 14:01:31.635363 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.634998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54572)
0927 14:01:31.635188 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.635261 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 14:01:31.635649 12708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33505: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
I20260927 14:01:31.640347 12715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:43407,127.22.26.253:33023,127.22.26.252:33505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.252.keytab
--principal=kudu/127.22.26.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.252:33505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.26.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:31.641800 12715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:31.643182 12715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:01:31.651791 12634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } attempt: 3
I20260927 14:01:31.652971 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.652391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54584)
0927 14:01:31.652638 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.652737 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 14:01:31.654335 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.653864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54590)
0927 14:01:31.654147 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.654232 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260927 14:01:31.654704 12634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:33505: Network error: Client connection negotiation failed: client connection to 127.22.26.252:33505: connect: Connection refused (error 111)
W20260927 14:01:31.654932 12721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:31.655064 12723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:31.655364 12720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:31.655984 12715 server_base.cc:1034] running on GCE node
I20260927 14:01:31.656771 12715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:31.658123 12715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:31.659379 12715 hybrid_clock.cc:648] HybridClock initialized: now 1790517691659304 us; error 39 us; skew 500 ppm
I20260927 14:01:31.659729 12715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:31 dist-test-slave-gkks krb5kdc[12554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517691, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:31.662727 12715 init.cc:377] Logged in from keytab as kudu/127.22.26.252@KRBTEST.COM (short username kudu)
I20260927 14:01:31.664400 12715 webserver.cc:460] Webserver started at http://127.22.26.252:46373/ using document root <none> and password file <none>
I20260927 14:01:31.665079 12715 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:31.665210 12715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:31.665530 12715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:31.668617 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/data/instance:
uuid: "1f83a8bf2da84a7aa6788716e94a2a3d"
format_stamp: "Formatted at 2026-09-27 14:01:31 on dist-test-slave-gkks"
server_key: "f9e52c0280717cf50e139eeee4d75d11"
server_key_iv: "bda88d85d2174b3b4958b843af8507d9"
server_key_version: "encryptionkey@0"
I20260927 14:01:31.669435 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/wal/instance:
uuid: "1f83a8bf2da84a7aa6788716e94a2a3d"
format_stamp: "Formatted at 2026-09-27 14:01:31 on dist-test-slave-gkks"
server_key: "f9e52c0280717cf50e139eeee4d75d11"
server_key_iv: "bda88d85d2174b3b4958b843af8507d9"
server_key_version: "encryptionkey@0"
I20260927 14:01:31.674443 12715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 14:01:31.677780 12730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:31.679242 12715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:01:31.679419 12715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/wal
uuid: "1f83a8bf2da84a7aa6788716e94a2a3d"
format_stamp: "Formatted at 2026-09-27 14:01:31 on dist-test-slave-gkks"
server_key: "f9e52c0280717cf50e139eeee4d75d11"
server_key_iv: "bda88d85d2174b3b4958b843af8507d9"
server_key_version: "encryptionkey@0"
I20260927 14:01:31.679641 12715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:31.696382 12715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:31.701074 12715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:31.701346 12715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:31.719864 12715 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.252:33505
I20260927 14:01:31.719895 12781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.252:33505 every 8 connection(s)
I20260927 14:01:31.721130 12715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/data/info.pb
I20260927 14:01:31.722661 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 12715
I20260927 14:01:31.722839 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790517330258165-22635-0/minicluster-data/master-2/wal/instance
I20260927 14:01:31.723351 22635 external_mini_cluster.cc:1428] Setting key d3cf0628aa5b56df2439b4c4cefd773b
I20260927 14:01:31.725221 12782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:31.733455 12782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:01:31 dist-test-slave-gkks krb5kdc[12554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.252@KRBTEST.COM
Sep 27 14:01:31 dist-test-slave-gkks krb5kdc[12554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517691, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.252@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:01:31.745502 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.725010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.22.26.252:33505)
0927 14:01:31.725769 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.725779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.725817 (+ 38us) server_negotiation.cc:407] Connection header received
0927 14:01:31.727393 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.727411 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.727775 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.728078 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.729910 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.731320 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.733082 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.733743 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.733872 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.738482 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:31.738489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:31.738498 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:31.738537 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:31.741363 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.742034 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.742045 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.742053 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.742251 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.742617 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.742629 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.742632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.743197 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:31.743512 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:31.743912 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 14:01:31.744323 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":356,"threads_started":1}
I20260927 14:01:31.751654 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.736346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:43407 (local address 127.0.0.1:55662)
0927 14:01:31.737079 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.737124 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:31.737298 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:31.737660 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:31.737666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:31.738234 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:31.738500 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.738513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.739549 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.739555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.740634 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.740645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.741469 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.741478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.741652 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.742459 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:31.742475 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:31.744617 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:31.748291 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.748305 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.748314 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.748970 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.749569 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.749586 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.749590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.749732 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.750362 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:31.750370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:31.750644 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 14:01:31.750835 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 14:01:31.751186 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":300,"threads_started":1}
I20260927 14:01:31.751754 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.736419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.22.26.254:43407)
0927 14:01:31.736669 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.736675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.737223 (+ 548us) server_negotiation.cc:407] Connection header received
0927 14:01:31.737438 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.737442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.737501 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.737605 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.738652 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.739398 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.740831 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.741254 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.741397 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.744811 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:31.744820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:31.744824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:31.744863 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:31.748129 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.749125 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.749131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.749134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.749334 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.749864 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.749870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.749873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.750221 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:31.750394 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:31.750970 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 14:01:31.751251 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":63}
I20260927 14:01:31.753115 12782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:01:31 dist-test-slave-gkks krb5kdc[12554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
Sep 27 14:01:31 dist-test-slave-gkks krb5kdc[12554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517691, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.252@KRBTEST.COM for kudu/127.22.26.253@KRBTEST.COM
I20260927 14:01:31.765431 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.749163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55678 (local address 127.22.26.254:43407)
0927 14:01:31.749725 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.749729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.749747 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:01:31.749792 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.749795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.749842 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.749957 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.751194 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.752303 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.753492 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.753971 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.754084 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.759111 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:31.759120 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:31.759124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:31.759164 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:31.762376 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.763234 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.763241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.763245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.763388 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.763810 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.763815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.763817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.764103 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:31.764428 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:31.764759 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 14:01:31.764995 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":263,"threads_started":1}
I20260927 14:01:31.768404 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.753926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:33023 (local address 127.0.0.1:34254)
0927 14:01:31.754249 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.754264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:31.754355 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:31.754712 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:31.754718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:31.755218 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:31.755363 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.755373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.756808 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.756813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.758341 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.758359 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.759364 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.759369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.759497 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.760138 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:31.760166 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:31.762708 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:31.766024 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.766028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.766030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.766392 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.766783 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.766787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.766789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.766931 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.767637 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:31.767641 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:31.767781 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 14:01:31.767918 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 14:01:31.768129 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260927 14:01:31.768711 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.753926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.22.26.253:33023)
0927 14:01:31.754133 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.754142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.754346 (+ 204us) server_negotiation.cc:407] Connection header received
0927 14:01:31.754457 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.754461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.754543 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.754656 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.755517 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.756647 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.758539 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.759157 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.759326 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.762912 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:31.762916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:31.762919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:31.762951 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:31.765899 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.766492 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.766496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.766499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.766666 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.767061 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.767068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.767071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.767462 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:31.767625 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:31.768032 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 14:01:31.768294 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:01:31.769755 12782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:01:31 dist-test-slave-gkks krb5kdc[12554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.253@KRBTEST.COM
Sep 27 14:01:31 dist-test-slave-gkks krb5kdc[12554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517691, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.252@KRBTEST.COM for kudu/127.22.26.252@KRBTEST.COM
I20260927 14:01:31.784603 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.770606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54604)
0927 14:01:31.770744 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.770758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:31.770881 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:31.771338 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:31.771343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:31.771805 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:31.771965 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.771975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.773230 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.773236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.774563 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.774579 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.775401 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.775408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.775646 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.776192 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:31.776215 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:31.778393 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:31.781993 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.781996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.781999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.782333 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.782675 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.782680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.782683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.782838 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.783497 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:31.783501 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:31.783650 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 14:01:31.783800 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 14:01:31.784054 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 14:01:31.784690 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.770882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54604 (local address 127.22.26.252:33505)
0927 14:01:31.771025 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.771029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.771050 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:01:31.771137 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.771143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.771208 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.771286 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.772117 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.773091 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.774716 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.775235 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.775346 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.778571 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:31.778577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:31.778581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:31.778615 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:31.781850 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.782429 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.782433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.782435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.782547 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.782958 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.782961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.782963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.783365 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:31.783488 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:31.784065 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 14:01:31.784321 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 14:01:31.785251 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.767175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.22.26.253:33023)
0927 14:01:31.767781 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.767786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.767812 (+ 26us) server_negotiation.cc:407] Connection header received
0927 14:01:31.767846 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.767849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.767924 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.768022 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.769628 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.770764 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.772467 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.773093 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.773224 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.778345 (+ 5121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:31.778350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:31.778354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:31.778383 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:31.781866 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.783123 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.783129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.783132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.783273 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.783693 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.783696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.783698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.784057 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:31.784337 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:31.784635 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 14:01:31.784844 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":281,"threads_started":1}
I20260927 14:01:31.790165 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:31.790822 12782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d: Bootstrap starting.
I20260927 14:01:31.794601 12782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:31.795722 12782 log.cc:824] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:31.794198 12609 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d359099e82124452bc49adb2ad0f4e1c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55678
I20260927 14:01:31.798632 12782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d: No bootstrap required, opened a new log
I20260927 14:01:31.805651 12782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } }
I20260927 14:01:31.805970 12782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f83a8bf2da84a7aa6788716e94a2a3d, State: Initialized, Role: FOLLOWER
I20260927 14:01:31.806416 12782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } }
I20260927 14:01:31.807273 12790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } } }
I20260927 14:01:31.807611 12790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:31.808504 12782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d [sys.catalog]: configured and running, proceeding with master startup.
W20260927 14:01:31.817519 12801 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:01:31.817669 12801 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:01:31.815623 12715 master_runner.cc:186] Error getting master registration for 127.22.26.254:43407: OK, instance_id { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" instance_seqno: 1790517690692914 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 14:01:31.826699 12715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 14:01:31.831938 12634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } attempt: 4
Sep 27 14:01:31 dist-test-slave-gkks krb5kdc[12554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517690, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for kudu/127.22.26.252@KRBTEST.COM
I20260927 14:01:31.847800 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.832581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54612)
0927 14:01:31.832800 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.832823 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:31.832948 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:31.833345 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:31.833352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:31.833879 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:31.834091 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.834103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.835518 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.835524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.837048 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.837065 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.838050 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.838057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.838252 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.839036 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:31.839060 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:31.841793 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:31.845269 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.845274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.845279 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.845656 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.846025 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.846031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.846035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.846178 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.846773 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:31.846778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:31.846935 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 14:01:31.847079 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 14:01:31.847373 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 14:01:31.847802 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.832671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.22.26.252:33505)
0927 14:01:31.832865 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.832873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.832900 (+ 27us) server_negotiation.cc:407] Connection header received
0927 14:01:31.833087 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.833094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.833183 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.833295 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.834236 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.835360 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.837193 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.837889 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.838016 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.842014 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:31.842021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:31.842026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:31.842063 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:31.845117 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.845779 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.845785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.845788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.845912 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.846288 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.846294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.846297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.846635 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:31.846799 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:31.847228 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 14:01:31.847423 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:01:31.852285 12634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c: Bootstrap starting.
I20260927 14:01:31.855419 12634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:31.856401 12634 log.cc:824] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:31.859305 12634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c: No bootstrap required, opened a new log
I20260927 14:01:31.864560 12634 raft_consensus.cc:348] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } }
I20260927 14:01:31.864893 12634 raft_consensus.cc:729] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d359099e82124452bc49adb2ad0f4e1c, State: Initialized, Role: FOLLOWER
I20260927 14:01:31.865275 12634 consensus_queue.cc:260] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } }
I20260927 14:01:31.866200 12804 sys_catalog.cc:455] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } } }
I20260927 14:01:31.866565 12804 sys_catalog.cc:458] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:31.867421 12634 sys_catalog.cc:564] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:31.867563 12609 raft_consensus.cc:482] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 14:01:31.867774 12609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:31.871438 12609 raft_consensus.cc:504] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } }
I20260927 14:01:31.874498 12757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d359099e82124452bc49adb2ad0f4e1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d"
I20260927 14:01:31.875012 12757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:31.878456 12757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d359099e82124452bc49adb2ad0f4e1c in term 1.
I20260927 14:01:31.879551 12586 leader_election.cc:302] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f83a8bf2da84a7aa6788716e94a2a3d, d359099e82124452bc49adb2ad0f4e1c; no voters:
I20260927 14:01:31.880034 12804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:31.886029 12609 leader_election.cc:288] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [CANDIDATE]: Term 1 election: Requested vote from peers 73169e62ad3a4e75b4894ea8b777e253 (127.22.26.253:33023), 1f83a8bf2da84a7aa6788716e94a2a3d (127.22.26.252:33505)
W20260927 14:01:31.890339 12815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:01:31.890499 12815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:01:31.890872 12804 raft_consensus.cc:686] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [term 1 LEADER]: Becoming Leader. State: Replica: d359099e82124452bc49adb2ad0f4e1c, State: Running, Role: LEADER
I20260927 14:01:31.891458 12804 consensus_queue.cc:237] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } }
I20260927 14:01:31.903680 12817 sys_catalog.cc:455] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d359099e82124452bc49adb2ad0f4e1c. Latest consensus state: current_term: 1 leader_uuid: "d359099e82124452bc49adb2ad0f4e1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } } }
I20260927 14:01:31.904150 12817 sys_catalog.cc:458] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:31.911712 12821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:31.914053 12821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:31.907891 12567 master_runner.cc:186] Error getting master registration for 127.22.26.253:33023: OK, instance_id { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" instance_seqno: 1790517691208594 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260927 14:01:31.932866 12588 tablet.cc:2369] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:01:31.934849 12708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } attempt: 5
I20260927 14:01:31.936429 12567 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 14:01:31 dist-test-slave-gkks krb5kdc[12554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517691, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.253@KRBTEST.COM for kudu/127.22.26.252@KRBTEST.COM
W20260927 14:01:31.953511 12736 tablet.cc:2369] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:01:31.974385 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.935513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:33505 (local address 127.0.0.1:54622)
0927 14:01:31.935932 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:31.935954 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:31.936077 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:31.937165 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:31.937171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:31.937706 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:31.937921 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.937933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.939519 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.939524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.941012 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:31.941028 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.942364 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.942370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:31.942662 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.943468 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:31.943495 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:31.946844 (+ 3349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:31.952598 (+ 5754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.952634 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.952637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.955778 (+ 3141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.956712 (+ 934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:31.956746 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:31.956753 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:31.957157 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:31.958550 (+ 1393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:31.958583 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:31.958934 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 14:01:31.965632 (+ 6698us) client_negotiation.cc:240] Negotiation successful
0927 14:01:31.969228 (+ 3596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260927 14:01:31.976688 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.936382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.22.26.252:33505)
0927 14:01:31.936580 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.936586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.936606 (+ 20us) server_negotiation.cc:407] Connection header received
0927 14:01:31.936751 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.936757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.936837 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.936943 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.938119 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.939315 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.941250 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.941990 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.942112 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.947154 (+ 5042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:31.947162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:31.947167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:31.947209 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:31.952366 (+ 5157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.956023 (+ 3657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.956060 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.956067 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.956475 (+ 408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.957372 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.957380 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.957414 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.958306 (+ 892us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:31.967859 (+ 9553us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:31.969296 (+ 1437us) server_negotiation.cc:299] Negotiation successful
0927 14:01:31.971827 (+ 2531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 14:01:31.983266 12708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253: Bootstrap starting.
I20260927 14:01:31.984683 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.952973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.22.26.254:43407)
0927 14:01:31.953188 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.953195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.965457 (+ 12262us) server_negotiation.cc:407] Connection header received
0927 14:01:31.965938 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.965944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.966049 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.966135 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.968784 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.969903 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.972196 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.973123 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.973245 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.977585 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:31.977591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:31.977596 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:31.977640 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:31.981374 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.982224 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.982229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.982234 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.982383 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:31.982837 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:31.982842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:31.982846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:31.983240 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:31.983475 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:31.984070 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 14:01:31.984333 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 14:01:31.989393 12708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:31.999919 12708 log.cc:824] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:32.004884 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.953082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.22.26.253:33023)
0927 14:01:31.953464 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.953471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.977671 (+ 24200us) server_negotiation.cc:407] Connection header received
0927 14:01:31.987114 (+ 9443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.987121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.987204 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.987283 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.989610 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.990669 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.992583 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:31.993482 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:31.993604 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:31.996968 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:31.996975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:31.996980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:31.997020 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:32.001651 (+ 4631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:32.002523 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:32.002529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:32.002533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:32.002673 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:32.003136 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:32.003141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:32.003145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:32.003487 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:32.003736 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:32.004311 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 14:01:32.004537 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260927 14:01:32.014868 12708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253: No bootstrap required, opened a new log
I20260927 14:01:32.019004 12757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d [term 1 FOLLOWER]: Refusing update from remote peer d359099e82124452bc49adb2ad0f4e1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:01:32.020462 12804 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:01:32.023259 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:31.959343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.22.26.252:33505)
0927 14:01:31.990930 (+ 31587us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:31.991115 (+ 185us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:31.991135 (+ 20us) server_negotiation.cc:407] Connection header received
0927 14:01:31.991463 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:31.991764 (+ 301us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:31.992266 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:31.995493 (+ 3227us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:31.996230 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:32.000083 (+ 3853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:32.002312 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:32.005298 (+ 2986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:32.006741 (+ 1443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:32.014020 (+ 7279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:32.014029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:32.014035 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:32.014080 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:32.019247 (+ 5167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:32.020187 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:32.020192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:32.020196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:32.020349 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:32.020801 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:32.020806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:32.020811 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:32.021200 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:32.022412 (+ 1212us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:32.022538 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 14:01:32.022778 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30866,"thread_start_us":33867,"threads_started":1}
I20260927 14:01:32.039525 12708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } }
I20260927 14:01:32.040365 12708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73169e62ad3a4e75b4894ea8b777e253, State: Initialized, Role: FOLLOWER
I20260927 14:01:32.041237 12708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } }
I20260927 14:01:32.050251 12790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d359099e82124452bc49adb2ad0f4e1c. Latest consensus state: current_term: 1 leader_uuid: "d359099e82124452bc49adb2ad0f4e1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } } }
I20260927 14:01:32.050575 12790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:32.052155 12683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d359099e82124452bc49adb2ad0f4e1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "73169e62ad3a4e75b4894ea8b777e253"
I20260927 14:01:32.052510 12682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:32.056473 12823 mvcc.cc:205] Tried to move back new op lower bound from 7333960466483912704 to 7333960465993125888. Current Snapshot: MvccSnapshot[applied={T|T < 7333960466483912704}]
I20260927 14:01:32.057108 12682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253 [term 1 FOLLOWER]: Refusing update from remote peer d359099e82124452bc49adb2ad0f4e1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:01:32.057754 12708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:32.060735 12683 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d359099e82124452bc49adb2ad0f4e1c in term 1.
I20260927 14:01:32.063437 12804 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:01:32.063601 12832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } } }
I20260927 14:01:32.064021 12832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:32.071548 12830 mvcc.cc:205] Tried to move back new op lower bound from 7333960466483912704 to 7333960465993125888. Current Snapshot: MvccSnapshot[applied={T|T < 7333960466483912704}]
I20260927 14:01:32.073614 12790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d359099e82124452bc49adb2ad0f4e1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } } }
I20260927 14:01:32.073889 12790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:32.082029 12804 sys_catalog.cc:455] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d359099e82124452bc49adb2ad0f4e1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } } }
I20260927 14:01:32.082434 12804 sys_catalog.cc:458] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:32.083976 12835 mvcc.cc:205] Tried to move back new op lower bound from 7333960466483912704 to 7333960465993125888. Current Snapshot: MvccSnapshot[applied={T|T < 7333960466483912704}]
I20260927 14:01:32.084904 12821 catalog_manager.cc:1347] Generated new cluster ID: 8eadf24fed6142bf95cb341becc773cf
I20260927 14:01:32.085003 12821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:32.111526 12838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d359099e82124452bc49adb2ad0f4e1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } } }
I20260927 14:01:32.112103 12838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:32.112969 12832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d359099e82124452bc49adb2ad0f4e1c. Latest consensus state: current_term: 1 leader_uuid: "d359099e82124452bc49adb2ad0f4e1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } } }
I20260927 14:01:32.113343 12832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:32.170068 12817 sys_catalog.cc:455] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d359099e82124452bc49adb2ad0f4e1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d359099e82124452bc49adb2ad0f4e1c" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 43407 } } peers { permanent_uuid: "73169e62ad3a4e75b4894ea8b777e253" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 33023 } } peers { permanent_uuid: "1f83a8bf2da84a7aa6788716e94a2a3d" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 33505 } } }
I20260927 14:01:32.170368 12817 sys_catalog.cc:458] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:32.183387 12821 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:32.185456 12821 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:32.202098 12849 catalog_manager.cc:1259] Loaded cluster ID: 8eadf24fed6142bf95cb341becc773cf
I20260927 14:01:32.205013 12849 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253: loading cluster ID for follower catalog manager: success
I20260927 14:01:32.216648 12849 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253: acquiring CA information for follower catalog manager: success
W20260927 14:01:32.218309 12849 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260927 14:01:32.225281 12821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d359099e82124452bc49adb2ad0f4e1c: Generated new TSK 0
I20260927 14:01:32.226492 12821 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 14:01:32.364065 12787 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:32.337927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.22.26.254:43407)
0927 14:01:32.338151 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:32.338158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:32.361341 (+ 23183us) server_negotiation.cc:407] Connection header received
0927 14:01:32.362362 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:32.362368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:32.362467 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:32.362547 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:32.363782 (+ 1235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55696: BlockingRecv error: recv got EOF from 127.0.0.1:55696 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 14:01:32.369112 12785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:32.356395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.22.26.252:33505)
0927 14:01:32.356611 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:32.356656 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:32.366997 (+ 10341us) server_negotiation.cc:407] Connection header received
0927 14:01:32.367512 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:32.367522 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:32.367611 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:32.367687 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:32.368836 (+ 1149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54636: BlockingRecv error: recv got EOF from 127.0.0.1:54636 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 14:01:32.371013 12788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:32.344616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.22.26.253:33023)
0927 14:01:32.344802 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:32.344809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:32.366414 (+ 21605us) server_negotiation.cc:407] Connection header received
0927 14:01:32.369480 (+ 3066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:32.369490 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:32.369598 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:32.369674 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:32.370769 (+ 1095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34282: BlockingRecv error: recv got EOF from 127.0.0.1:34282 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 14:01:32.416034 12788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:32.402925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.22.26.253:33023)
0927 14:01:32.403148 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:32.403154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:32.413815 (+ 10661us) server_negotiation.cc:407] Connection header received
0927 14:01:32.414399 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:32.414406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:32.414559 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:32.414645 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:32.415772 (+ 1127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34284: BlockingRecv error: recv got EOF from 127.0.0.1:34284 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 14:01:32.447520 12788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:32.445026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.22.26.253:33023)
0927 14:01:32.445255 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:32.445262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:32.445825 (+ 563us) server_negotiation.cc:407] Connection header received
0927 14:01:32.446063 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:32.446072 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:32.446169 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:32.446291 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:32.447221 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34300: BlockingRecv error: recv got EOF from 127.0.0.1:34300 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 14:01:32.454968 12788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:32.452757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.22.26.253:33023)
0927 14:01:32.452925 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:32.452931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:32.453416 (+ 485us) server_negotiation.cc:407] Connection header received
0927 14:01:32.453656 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:32.453662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:32.453748 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:32.453858 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:32.454704 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34302: BlockingRecv error: recv got EOF from 127.0.0.1:34302 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 14:01:32.464730 12788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:32.462825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34310 (local address 127.22.26.253:33023)
0927 14:01:32.463026 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:32.463033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:32.463155 (+ 122us) server_negotiation.cc:407] Connection header received
0927 14:01:32.463470 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:32.463477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:32.463568 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:32.463681 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:32.464470 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34310: BlockingRecv error: recv got EOF from 127.0.0.1:34310 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 14:01:32.483520 12788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:32.481171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.22.26.253:33023)
0927 14:01:32.481363 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:32.481369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:32.481865 (+ 496us) server_negotiation.cc:407] Connection header received
0927 14:01:32.482068 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:32.482074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:32.482186 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:32.482295 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:32.483268 (+ 973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34318: BlockingRecv error: recv got EOF from 127.0.0.1:34318 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 14:01:32.505839 12788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:32.503475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.22.26.253:33023)
0927 14:01:32.503688 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:32.503694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:32.504250 (+ 556us) server_negotiation.cc:407] Connection header received
0927 14:01:32.504457 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:32.504463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:32.504558 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:32.504692 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:32.505561 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34334: BlockingRecv error: recv got EOF from 127.0.0.1:34334 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 14:01:32.548053 12788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:32.545776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34340 (local address 127.22.26.253:33023)
0927 14:01:32.545973 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:32.545979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:32.546482 (+ 503us) server_negotiation.cc:407] Connection header received
0927 14:01:32.546702 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:32.546708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:32.546804 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:32.546929 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:32.547805 (+ 876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34340: BlockingRecv error: recv got EOF from 127.0.0.1:34340 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 14:01:32.617955 12788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:32.615584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.22.26.253:33023)
0927 14:01:32.615859 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:32.615870 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:32.616255 (+ 385us) server_negotiation.cc:407] Connection header received
0927 14:01:32.616497 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:32.616501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:32.616597 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:32.616684 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:32.617642 (+ 958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34354: BlockingRecv error: recv got EOF from 127.0.0.1:34354 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260927 14:01:32.754400 12788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:32.752517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34356 (local address 127.22.26.253:33023)
0927 14:01:32.752745 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:32.752753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:32.753032 (+ 279us) server_negotiation.cc:407] Connection header received
0927 14:01:32.753277 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:32.753283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:32.753354 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:32.753467 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:32.754161 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34356: BlockingRecv error: recv got EOF from 127.0.0.1:34356 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 14:01:32.819720 12801 catalog_manager.cc:1259] Loaded cluster ID: 8eadf24fed6142bf95cb341becc773cf
I20260927 14:01:32.819885 12801 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d: loading cluster ID for follower catalog manager: success
I20260927 14:01:32.822422 12801 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d: acquiring CA information for follower catalog manager: success
I20260927 14:01:32.824237 12801 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1f83a8bf2da84a7aa6788716e94a2a3d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 14:01:33.016713 12788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:33.014171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.22.26.253:33023)
0927 14:01:33.014399 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:33.014407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:33.015003 (+ 596us) server_negotiation.cc:407] Connection header received
0927 14:01:33.015378 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:33.015382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:33.015493 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:33.015604 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:33.016510 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34364: BlockingRecv error: recv got EOF from 127.0.0.1:34364 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 14:01:33.221084 12849 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 73169e62ad3a4e75b4894ea8b777e253: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 14:01:33.542553 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:33.536601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.22.26.253:33023)
0927 14:01:33.537360 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:33.537366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:33.537391 (+ 25us) server_negotiation.cc:407] Connection header received
0927 14:01:33.537457 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:33.537461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:33.537616 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:33.537727 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:33.538181 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:33.538946 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:33.540321 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:33.540735 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:33.540830 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:33.541354 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:33.541905 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:33.542030 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:33.542116 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 14:01:33.542191 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":553,"threads_started":1}
I20260927 14:01:33.554591 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:33.549243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.22.26.252:33505)
0927 14:01:33.549743 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:33.549747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:33.549765 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:01:33.549949 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:33.549953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:33.550110 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:33.550257 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:33.550635 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:33.551371 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:33.552621 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:33.553030 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:33.553132 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:33.553570 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:33.554098 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:33.554205 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:33.554274 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 14:01:33.554339 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":246,"threads_started":1}
I20260927 14:01:33.575682 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:33.562488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.22.26.253:33023)
0927 14:01:33.562682 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:33.562688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:33.569242 (+ 6554us) server_negotiation.cc:407] Connection header received
0927 14:01:33.569537 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:33.569542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:33.569627 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:33.569690 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:33.570491 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:33.571530 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:33.573809 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:33.574306 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:33.574454 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:33.574915 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:33.575063 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:33.575170 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:33.575288 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 14:01:33.575369 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:01:33.576032 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:33.562402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.22.26.252:33505)
0927 14:01:33.567917 (+ 5515us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:33.567926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:33.569827 (+ 1901us) server_negotiation.cc:407] Connection header received
0927 14:01:33.570195 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:33.570201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:33.570282 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:33.570352 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:33.571316 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:33.572386 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:33.574026 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:33.574634 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:33.574751 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:33.575230 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:33.575357 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:33.575420 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:33.575683 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 14:01:33.575760 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5332}
I20260927 14:01:33.577090 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:33.562241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.22.26.254:43407)
0927 14:01:33.568868 (+ 6627us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:33.568875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:33.568896 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:01:33.569024 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:33.569030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:33.569252 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:33.569449 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:33.570112 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:33.571270 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:33.573827 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:33.574437 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:33.574552 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:33.575246 (+ 694us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:33.576019 (+ 773us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:33.576291 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:33.576743 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 14:01:33.576832 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6500,"thread_start_us":6408,"threads_started":1}
I20260927 14:01:33.581648 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 12567
I20260927 14:01:33.583698 12629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:01:33.701114 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 12567
I20260927 14:01:33.723306 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 12641
I20260927 14:01:33.724455 12703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:01:33.834484 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 12641
I20260927 14:01:33.856894 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 12715
I20260927 14:01:33.858196 12777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:01:33.966310 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 12715
2026-09-27T14:01:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5090 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:01:34 dist-test-slave-gkks krb5kdc[12935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:01:34 dist-test-slave-gkks krb5kdc[12935](info): set up 2 sockets
Sep 27 14:01:34 dist-test-slave-gkks krb5kdc[12935](info): commencing operation
krb5kdc: starting...
W20260927 14:01:35.083945 22635 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/test-user.keytab.
Sep 27 14:01:35 dist-test-slave-gkks krb5kdc[12935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:01:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:01:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
Entry for principal kudu/127.22.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab.
I20260927 14:01:35.350384 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:37917
--webserver_interface=127.22.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:37917,127.22.26.253:43051,127.22.26.252:37853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:01:35.685395 12948 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 14:01:35.685767 12948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:35.685895 12948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:35.686031 12948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:35.693809 12948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:35.693900 12948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:35.694005 12948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:35.694098 12948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:35.694206 12948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:35.704002 12948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:37917,127.22.26.253:43051,127.22.26.252:37853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.254.keytab
--principal=kudu/127.22.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.254:37917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:35.705327 12948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:35.706804 12948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:35.719249 12956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:35.719174 12953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:35.719274 12954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:35.720378 12948 server_base.cc:1034] running on GCE node
I20260927 14:01:35.721500 12948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:35.723014 12948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:35.724289 12948 hybrid_clock.cc:648] HybridClock initialized: now 1790517695724216 us; error 53 us; skew 500 ppm
I20260927 14:01:35.724619 12948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:35 dist-test-slave-gkks krb5kdc[12935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517695, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:35.727878 12948 init.cc:377] Logged in from keytab as kudu/127.22.26.254@KRBTEST.COM (short username kudu)
I20260927 14:01:35.729319 12948 webserver.cc:460] Webserver started at http://127.22.26.254:44925/ using document root <none> and password file <none>
I20260927 14:01:35.730015 12948 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:35.730149 12948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:35.730479 12948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:35.733495 12948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/data/instance:
uuid: "fdc84d3e979a41a6805f38e94db607f0"
format_stamp: "Formatted at 2026-09-27 14:01:35 on dist-test-slave-gkks"
server_key: "25d370b11530d72e4b90707dbd11d5ae"
server_key_iv: "535909ca5781a474bbbe539dd4f87463"
server_key_version: "encryptionkey@0"
I20260927 14:01:35.734316 12948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/wal/instance:
uuid: "fdc84d3e979a41a6805f38e94db607f0"
format_stamp: "Formatted at 2026-09-27 14:01:35 on dist-test-slave-gkks"
server_key: "25d370b11530d72e4b90707dbd11d5ae"
server_key_iv: "535909ca5781a474bbbe539dd4f87463"
server_key_version: "encryptionkey@0"
I20260927 14:01:35.739631 12948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 14:01:35.743047 12963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:35.744511 12948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:01:35.744761 12948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/wal
uuid: "fdc84d3e979a41a6805f38e94db607f0"
format_stamp: "Formatted at 2026-09-27 14:01:35 on dist-test-slave-gkks"
server_key: "25d370b11530d72e4b90707dbd11d5ae"
server_key_iv: "535909ca5781a474bbbe539dd4f87463"
server_key_version: "encryptionkey@0"
I20260927 14:01:35.745003 12948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:35.764096 12948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:35.768656 12948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:35.768968 12948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:35.787536 12948 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.254:37917
I20260927 14:01:35.787560 13014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.254:37917 every 8 connection(s)
I20260927 14:01:35.788951 12948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/data/info.pb
I20260927 14:01:35.792578 13015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:35.796590 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 12948
I20260927 14:01:35.796801 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-0/wal/instance
I20260927 14:01:35.797379 22635 external_mini_cluster.cc:1428] Setting key 0ff95a9b3f1afd0461ba5a57973bff84
I20260927 14:01:35.800858 13015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:01:35 dist-test-slave-gkks krb5kdc[12935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517695, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.26.253@KRBTEST.COM; defaulting to no policy
I20260927 14:01:35.826308 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:35.805843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51056 (local address 127.22.26.254:37917)
0927 14:01:35.806543 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:35.806554 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:35.806606 (+ 52us) server_negotiation.cc:407] Connection header received
0927 14:01:35.808432 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:35.808438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:35.808674 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:35.808867 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:35.810446 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:35.811723 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:35.813913 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:35.814468 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:35.814611 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:35.818709 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:35.818717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:35.818728 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:35.818760 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:35.821741 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:35.822450 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:35.822464 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:35.822467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:35.822611 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:35.823004 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:35.823012 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:35.823017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:35.823580 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:35.823800 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:35.824342 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 14:01:35.824726 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":482,"thread_start_us":317,"threads_started":1}
I20260927 14:01:35.826333 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:35.804515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:37917 (local address 127.0.0.1:51056)
0927 14:01:35.805333 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:35.805380 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:35.807939 (+ 2559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:35.808931 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:35.808940 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:35.809735 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:35.810229 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:35.810255 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:35.811883 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:35.811892 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:35.813761 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:35.813780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:35.814713 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:35.814722 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:35.814976 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:35.815780 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:35.815844 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:35.818491 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:35.821887 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:35.821900 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:35.821912 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:35.822308 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:35.822724 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:35.822729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:35.822733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:35.822896 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:35.823739 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:35.823752 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:35.823959 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 14:01:35.824222 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 14:01:35.824849 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":155,"thread_start_us":363,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.253@KRBTEST.COM" created.
I20260927 14:01:35.829653 13015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:01:35.830788 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:35.830385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43051 (local address 127.0.0.1:38048)
0927 14:01:35.830560 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:35.830670 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 14:01:35.831125 12964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
I20260927 14:01:35.832715 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:35.832364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43051 (local address 127.0.0.1:38060)
0927 14:01:35.832532 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:35.832596 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 14:01:35.833022 13015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:43051: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.253.keytab.
Entry for principal kudu/127.22.26.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.253.keytab.
I20260927 14:01:35.859055 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.253:43051
--webserver_interface=127.22.26.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.253.keytab
--principal=kudu/127.22.26.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:37917,127.22.26.253:43051,127.22.26.252:37853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:01:35.882388 13015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } attempt: 1
I20260927 14:01:35.883560 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:35.883074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43051 (local address 127.0.0.1:38068)
0927 14:01:35.883303 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:35.883432 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 14:01:35.884601 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:35.884258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43051 (local address 127.0.0.1:38076)
0927 14:01:35.884403 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:35.884477 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 14:01:35.884795 13015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:43051: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
I20260927 14:01:35.953064 13015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } attempt: 2
I20260927 14:01:35.954250 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:35.953746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43051 (local address 127.0.0.1:38084)
0927 14:01:35.954011 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:35.954102 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 14:01:35.955356 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:35.954947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43051 (local address 127.0.0.1:38092)
0927 14:01:35.955150 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:35.955224 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 14:01:35.955551 13015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:43051: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
I20260927 14:01:36.046838 13015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } attempt: 3
I20260927 14:01:36.048115 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.047651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43051 (local address 127.0.0.1:38100)
0927 14:01:36.047880 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.047980 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 14:01:36.049315 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.048953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43051 (local address 127.0.0.1:38114)
0927 14:01:36.049111 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.049177 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 14:01:36.049507 13015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:43051: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
W20260927 14:01:36.190126 13022 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 14:01:36.190517 13022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:36.190642 13022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:36.190779 13022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 14:01:36.192764 13015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } attempt: 4
I20260927 14:01:36.194190 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.193640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43051 (local address 127.0.0.1:38124)
0927 14:01:36.193960 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.194047 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260927 14:01:36.195348 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.194939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43051 (local address 127.0.0.1:38136)
0927 14:01:36.195121 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.195221 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 14:01:36.195544 13015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.253:43051: Network error: Client connection negotiation failed: client connection to 127.22.26.253:43051: connect: Connection refused (error 111)
W20260927 14:01:36.198778 13022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:36.198908 13022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:36.199023 13022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:36.199126 13022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:36.199242 13022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:36.209432 13022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:37917,127.22.26.253:43051,127.22.26.252:37853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.253.keytab
--principal=kudu/127.22.26.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.253:43051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.26.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:36.210801 13022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:36.212180 13022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:36.224268 13027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:36.224345 13030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:36.224378 13028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:36.225363 13022 server_base.cc:1034] running on GCE node
I20260927 14:01:36.226218 13022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:36.227705 13022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:36.228991 13022 hybrid_clock.cc:648] HybridClock initialized: now 1790517696228882 us; error 59 us; skew 500 ppm
I20260927 14:01:36.229421 13022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:36 dist-test-slave-gkks krb5kdc[12935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517696, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:36.232625 13022 init.cc:377] Logged in from keytab as kudu/127.22.26.253@KRBTEST.COM (short username kudu)
I20260927 14:01:36.234169 13022 webserver.cc:460] Webserver started at http://127.22.26.253:33065/ using document root <none> and password file <none>
I20260927 14:01:36.234880 13022 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:36.235020 13022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:36.235379 13022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:36.238339 13022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/data/instance:
uuid: "0f304acaaea8481eae31fc4781565ef0"
format_stamp: "Formatted at 2026-09-27 14:01:36 on dist-test-slave-gkks"
server_key: "ac73443c58cf9c50c504efdf867993d9"
server_key_iv: "c9b545d4cc3a12d16015ea725a96f1e5"
server_key_version: "encryptionkey@0"
I20260927 14:01:36.239259 13022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/wal/instance:
uuid: "0f304acaaea8481eae31fc4781565ef0"
format_stamp: "Formatted at 2026-09-27 14:01:36 on dist-test-slave-gkks"
server_key: "ac73443c58cf9c50c504efdf867993d9"
server_key_iv: "c9b545d4cc3a12d16015ea725a96f1e5"
server_key_version: "encryptionkey@0"
I20260927 14:01:36.244578 13022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 14:01:36.248317 13037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:36.249835 13022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 14:01:36.250032 13022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/wal
uuid: "0f304acaaea8481eae31fc4781565ef0"
format_stamp: "Formatted at 2026-09-27 14:01:36 on dist-test-slave-gkks"
server_key: "ac73443c58cf9c50c504efdf867993d9"
server_key_iv: "c9b545d4cc3a12d16015ea725a96f1e5"
server_key_version: "encryptionkey@0"
I20260927 14:01:36.250249 13022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:36.261559 13022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:36.265364 13022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:36.265622 13022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:36.283548 13022 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.253:43051
I20260927 14:01:36.283560 13088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.253:43051 every 8 connection(s)
I20260927 14:01:36.284858 13022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/data/info.pb
I20260927 14:01:36.288462 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:36.294261 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 13022
I20260927 14:01:36.294447 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-1/wal/instance
I20260927 14:01:36.294919 22635 external_mini_cluster.cc:1428] Setting key 86596e1672e5b67aef2ec5f5ac53b9f3
I20260927 14:01:36.294848 13089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:01:36 dist-test-slave-gkks krb5kdc[12935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517696, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.253@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:01:36.317402 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.297564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.22.26.254:37917)
0927 14:01:36.297746 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:36.297751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:36.298980 (+ 1229us) server_negotiation.cc:407] Connection header received
0927 14:01:36.300891 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:36.300897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:36.300990 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:36.301104 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:36.302594 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.303457 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.305479 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.305897 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.305996 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.310227 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:36.310237 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:36.310245 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:36.310299 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:36.314020 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.314749 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.314758 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.314763 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.314913 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.315372 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.315380 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.315386 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.315794 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:36.315955 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:36.316671 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 14:01:36.316935 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:01:36.318059 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.298067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:37917 (local address 127.0.0.1:51066)
0927 14:01:36.298812 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.298854 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:36.300607 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:36.301342 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:36.301350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:36.301877 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:36.302415 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.302432 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.303647 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.303651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.305309 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.305327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.306124 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.306133 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.306479 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.307232 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:36.307270 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:36.309923 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:36.314162 (+ 4239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.314168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.314181 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.314618 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.315025 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.315028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.315030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.315263 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.315919 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:36.315924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:36.316267 (+ 343us) client_negotiation.cc:769] Sending connection context
0927 14:01:36.316514 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 14:01:36.316890 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":310,"threads_started":1}
I20260927 14:01:36.319933 13089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.26.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.26.252@KRBTEST.COM" created.
Sep 27 14:01:36 dist-test-slave-gkks krb5kdc[12935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517696, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.253@KRBTEST.COM for kudu/127.22.26.253@KRBTEST.COM
I20260927 14:01:36.334653 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.320812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43051 (local address 127.0.0.1:38146)
0927 14:01:36.320994 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.321023 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:36.321146 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:36.321891 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:36.321896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:36.322379 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:36.322503 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.322512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.323970 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.323978 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.324966 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.324978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.325768 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.325774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.325992 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.326576 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:36.326592 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:36.328843 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:36.332139 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.332145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.332150 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.332701 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.333082 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.333087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.333092 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.333283 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.333971 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:36.333975 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:36.334082 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 14:01:36.334195 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 14:01:36.334391 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":3}
I20260927 14:01:36.335167 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.320921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.22.26.253:43051)
0927 14:01:36.321496 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:36.321502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:36.321525 (+ 23us) server_negotiation.cc:407] Connection header received
0927 14:01:36.321564 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:36.321569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:36.321742 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:36.321904 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:36.322679 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.323838 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.325099 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.325623 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.325760 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.329043 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:36.329054 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:36.329067 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:36.329098 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:36.331994 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.332838 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.332845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.332847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.332963 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.333400 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.333403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.333405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.333845 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:36.334041 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:36.334383 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 14:01:36.334664 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":436,"thread_start_us":290,"threads_started":1}
I20260927 14:01:36.337738 13089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:01:36.338940 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.338514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60024)
0927 14:01:36.338686 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.338783 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 14:01:36.339223 13039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
I20260927 14:01:36.340790 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.340449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60038)
0927 14:01:36.340615 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.340697 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 14:01:36.341027 13089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:37853: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.26.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.252.keytab.
Entry for principal kudu/127.22.26.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.252.keytab.
I20260927 14:01:36.358207 22635 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
/tmp/dist-test-taskUeHuEC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.252:37853
--webserver_interface=127.22.26.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.252.keytab
--principal=kudu/127.22.26.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.26.212:44645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.26.254:37917,127.22.26.253:43051,127.22.26.252:37853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:01:36.390350 13089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } attempt: 1
I20260927 14:01:36.391786 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.391218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60042)
0927 14:01:36.391466 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.391597 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 14:01:36.392938 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.392529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60052)
0927 14:01:36.392711 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.392809 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 14:01:36.393194 13089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:37853: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
I20260927 14:01:36.461426 13089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } attempt: 2
I20260927 14:01:36.462728 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.462171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60064)
0927 14:01:36.462414 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.462533 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 14:01:36.463850 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.463458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60078)
0927 14:01:36.463612 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.463699 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 14:01:36.464118 13089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:37853: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
I20260927 14:01:36.494799 13015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } attempt: 5
Sep 27 14:01:36 dist-test-slave-gkks krb5kdc[12935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517695, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for kudu/127.22.26.253@KRBTEST.COM
I20260927 14:01:36.508998 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.495744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.22.26.253:43051)
0927 14:01:36.495953 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:36.495959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:36.495980 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:01:36.496143 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:36.496149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:36.496235 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:36.496348 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:36.497147 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.497914 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.499163 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.499946 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.500093 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.503444 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:36.503450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:36.503465 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:36.503509 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:36.506266 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.506912 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.506918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.506922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.507050 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.507467 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.507473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.507476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.507771 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:36.507935 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:36.508464 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 14:01:36.508679 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 14:01:36.509019 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.495610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43051 (local address 127.0.0.1:38150)
0927 14:01:36.495806 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.495855 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:36.495988 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:36.496377 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:36.496382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:36.496806 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:36.496988 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.496997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.498096 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.498102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.498977 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.498990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.500144 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.500179 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.500395 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.501045 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:36.501063 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:36.503174 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:36.506407 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.506412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.506417 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.506783 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.507187 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.507192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.507196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.507336 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.507897 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:36.507902 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:36.508191 (+ 289us) client_negotiation.cc:769] Sending connection context
0927 14:01:36.508348 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 14:01:36.508658 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 14:01:36.510416 13015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:01:36.511641 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.511176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60086)
0927 14:01:36.511370 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.511465 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 14:01:36.512627 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.512264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60096)
0927 14:01:36.512412 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.512507 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 14:01:36.512840 13015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:37853: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
I20260927 14:01:36.555361 13089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } attempt: 3
I20260927 14:01:36.556675 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.556027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60110)
0927 14:01:36.556309 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.556446 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 14:01:36.557917 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.557372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60122)
0927 14:01:36.557654 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.557753 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260927 14:01:36.558172 13089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:37853: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
I20260927 14:01:36.564065 13015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } attempt: 1
I20260927 14:01:36.565164 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.564652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60130)
0927 14:01:36.564850 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.564954 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 14:01:36.566222 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.565868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60136)
0927 14:01:36.566019 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.566102 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 14:01:36.566453 13015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:37853: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
I20260927 14:01:36.634708 13015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } attempt: 2
I20260927 14:01:36.636030 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.635420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60140)
0927 14:01:36.635696 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.635813 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 14:01:36.637168 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.636772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60142)
0927 14:01:36.636944 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.637047 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 14:01:36.637394 13015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:37853: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
W20260927 14:01:36.692309 13096 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 14:01:36.692615 13096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:01:36.692682 13096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:01:36.692772 13096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:01:36.699762 13096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:01:36.699846 13096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:01:36.699906 13096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:01:36.699956 13096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:01:36.700016 13096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:01:36.701381 13089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } attempt: 4
I20260927 14:01:36.702586 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.702004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60158)
0927 14:01:36.702273 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.702397 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 14:01:36.703681 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.703313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60168)
0927 14:01:36.703476 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.703556 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 14:01:36.703890 13089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:37853: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
I20260927 14:01:36.710498 13096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.26.212:44645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.26.254:37917,127.22.26.253:43051,127.22.26.252:37853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/krb5kdc/kudu_127.22.26.252.keytab
--principal=kudu/127.22.26.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.26.252:37853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.26.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 14:01:36.711798 13096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:01:36.713128 13096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:01:36.724431 13104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:36.724519 13101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:01:36.724493 13102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:01:36.725243 13096 server_base.cc:1034] running on GCE node
I20260927 14:01:36.725972 13096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:01:36.727322 13096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:01:36.728543 13096 hybrid_clock.cc:648] HybridClock initialized: now 1790517696728483 us; error 37 us; skew 500 ppm
I20260927 14:01:36.728823 13096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 14:01:36 dist-test-slave-gkks krb5kdc[12935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517696, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:01:36.731904 13096 init.cc:377] Logged in from keytab as kudu/127.22.26.252@KRBTEST.COM (short username kudu)
I20260927 14:01:36.733340 13096 webserver.cc:460] Webserver started at http://127.22.26.252:46341/ using document root <none> and password file <none>
I20260927 14:01:36.733995 13096 fs_manager.cc:359] Metadata directory not provided
I20260927 14:01:36.734109 13096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:01:36.734449 13096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:01:36.737381 13096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/data/instance:
uuid: "318369e0eabf4857b70f375e94638d07"
format_stamp: "Formatted at 2026-09-27 14:01:36 on dist-test-slave-gkks"
server_key: "66d6a0c09cc09c621757d8ee77068015"
server_key_iv: "648a164555e4c3ca1592c49a47e6b32c"
server_key_version: "encryptionkey@0"
I20260927 14:01:36.738166 13096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/wal/instance:
uuid: "318369e0eabf4857b70f375e94638d07"
format_stamp: "Formatted at 2026-09-27 14:01:36 on dist-test-slave-gkks"
server_key: "66d6a0c09cc09c621757d8ee77068015"
server_key_iv: "648a164555e4c3ca1592c49a47e6b32c"
server_key_version: "encryptionkey@0"
I20260927 14:01:36.743211 13096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 14:01:36.743621 13015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } attempt: 3
I20260927 14:01:36.744745 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.744188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60180)
0927 14:01:36.744417 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.744534 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 14:01:36.745882 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.745454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60186)
0927 14:01:36.745649 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.745743 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260927 14:01:36.746151 13015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.26.252:37853: Network error: Client connection negotiation failed: client connection to 127.22.26.252:37853: connect: Connection refused (error 111)
I20260927 14:01:36.747347 13111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:01:36.748744 13096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:01:36.748927 13096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/data,/tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/wal
uuid: "318369e0eabf4857b70f375e94638d07"
format_stamp: "Formatted at 2026-09-27 14:01:36 on dist-test-slave-gkks"
server_key: "66d6a0c09cc09c621757d8ee77068015"
server_key_iv: "648a164555e4c3ca1592c49a47e6b32c"
server_key_version: "encryptionkey@0"
I20260927 14:01:36.749136 13096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:01:36.763770 13096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:01:36.768090 13096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:01:36.768313 13096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:01:36.786103 13096 rpc_server.cc:307] RPC server started. Bound to: 127.22.26.252:37853
I20260927 14:01:36.786123 13162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.26.252:37853 every 8 connection(s)
I20260927 14:01:36.787576 13096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/data/info.pb
I20260927 14:01:36.792037 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:01:36.793740 22635 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu as pid 13096
I20260927 14:01:36.793962 22635 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUeHuEC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790517330258165-22635-0/minicluster-data/master-2/wal/instance
I20260927 14:01:36.794405 22635 external_mini_cluster.cc:1428] Setting key 4cfc8aeab6eab6483d7df2c45d2caa3f
I20260927 14:01:36.801748 13163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:01:36 dist-test-slave-gkks krb5kdc[12935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.252@KRBTEST.COM
Sep 27 14:01:36 dist-test-slave-gkks krb5kdc[12935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517696, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.252@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:01:36.819420 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.804685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.22.26.254:37917)
0927 14:01:36.804880 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:36.804886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:36.805616 (+ 730us) server_negotiation.cc:407] Connection header received
0927 14:01:36.805873 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:36.805879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:36.805979 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:36.806122 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:36.807129 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.807850 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.809700 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.810172 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.810282 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.813970 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:36.813975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:36.813978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:36.814009 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:36.816544 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.817207 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.817214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.817221 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.817366 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.817747 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.817754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.817759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.818016 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:36.818169 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:36.818764 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 14:01:36.819017 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 14:01:36.819911 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.796444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.22.26.252:37853)
0927 14:01:36.798977 (+ 2533us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:36.798998 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:36.799047 (+ 49us) server_negotiation.cc:407] Connection header received
0927 14:01:36.801362 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:36.801389 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:36.801915 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:36.802354 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:36.803853 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.805669 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.807169 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.807773 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.807957 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.812612 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:36.812620 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:36.812630 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:36.812658 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:36.815820 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.816473 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.816480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.816485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.816620 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.817006 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.817010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.817012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.817498 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:36.817844 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:36.818255 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 14:01:36.818651 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2263,"thread_start_us":1967,"threads_started":1}
I20260927 14:01:36.819957 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.804680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.254:37917 (local address 127.0.0.1:51072)
0927 14:01:36.805466 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.805505 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:36.805653 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:36.806163 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:36.806168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:36.806712 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:36.806944 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.806953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.808023 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.808028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.809515 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.809529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.810411 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.810419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.810620 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.811394 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:36.811422 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:36.813729 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:36.816677 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.816688 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.816696 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.817083 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.817492 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.817497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.817500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.817634 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.818152 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:36.818160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:36.818334 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 14:01:36.818510 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 14:01:36.818801 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":390,"threads_started":1}
I20260927 14:01:36.821527 13163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:01:36 dist-test-slave-gkks krb5kdc[12935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517696, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.252@KRBTEST.COM for kudu/127.22.26.253@KRBTEST.COM
Sep 27 14:01:36 dist-test-slave-gkks krb5kdc[12935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.254@KRBTEST.COM
I20260927 14:01:36.835245 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.822277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.253:43051 (local address 127.0.0.1:38154)
0927 14:01:36.822436 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.822454 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:36.822571 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:36.822961 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:36.822968 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:36.823455 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:36.823661 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.823673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.824621 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.824627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.825602 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.825615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.826351 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.826359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.826564 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.827213 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:36.827229 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:36.829425 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:36.832759 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.832765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.832769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.833140 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.833549 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.833554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.833558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.833703 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.834221 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:36.834226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:36.834393 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 14:01:36.834535 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 14:01:36.834824 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 14:01:36.835276 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.822285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.22.26.253:43051)
0927 14:01:36.822480 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:36.822489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:36.822511 (+ 22us) server_negotiation.cc:407] Connection header received
0927 14:01:36.822694 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:36.822699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:36.822811 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:36.822911 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:36.823820 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.824478 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.825771 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.826186 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.826306 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.829628 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:36.829638 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:36.829645 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:36.829693 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:36.832613 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.833265 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.833272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.833275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.833422 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.833815 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.833832 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.833839 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.834098 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:36.834258 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:36.834625 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 14:01:36.834851 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:01:36.836362 13163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:01:36.837623 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.822704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.22.26.254:37917)
0927 14:01:36.822878 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:36.822886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:36.823025 (+ 139us) server_negotiation.cc:407] Connection header received
0927 14:01:36.823212 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:36.823215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:36.823264 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:36.823343 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:36.824650 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.825333 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.826874 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.827409 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.827500 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.832067 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:36.832072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:36.832075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:36.832106 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:36.834669 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.835514 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.835522 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.835525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.835667 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.836123 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.836128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.836131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.836444 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:36.836716 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:36.837069 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 14:01:36.837300 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 14:01:36 dist-test-slave-gkks krb5kdc[12935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517696, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.252@KRBTEST.COM for kudu/127.22.26.252@KRBTEST.COM
Sep 27 14:01:36 dist-test-slave-gkks krb5kdc[12935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.26.253@KRBTEST.COM
I20260927 14:01:36.850407 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.837270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.22.26.252:37853)
0927 14:01:36.837405 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:36.837410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:36.837428 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:01:36.837503 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:36.837509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:36.837575 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:36.837672 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:36.838391 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.839189 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.840950 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.841655 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.841783 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.844963 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:36.844974 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:36.844977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:36.845002 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:36.847804 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.848498 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.848506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.848513 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.848649 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.849021 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.849030 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.849034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.849372 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:36.849475 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:36.849963 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 14:01:36.850168 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":35}
I20260927 14:01:36.850870 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.837101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60196)
0927 14:01:36.837288 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.837307 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:36.837411 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:36.837712 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:36.837716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:36.838128 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:36.838255 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.838263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.839655 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.839660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.840801 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.840814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.841800 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.841805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.841985 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.842542 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:36.842570 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:36.844786 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:36.847935 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.847939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.847942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.848347 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.848771 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.848775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.848777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.848909 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.849487 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:36.849493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:36.849649 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 14:01:36.849873 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 14:01:36.850355 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":124}
I20260927 14:01:36.855671 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.839278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.22.26.253:43051)
0927 14:01:36.839422 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:36.839429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:36.839446 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:01:36.839622 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:36.839626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:36.839694 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:36.839798 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:36.841356 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.842344 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.844150 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.844620 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.844756 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.849131 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:36.849136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:36.849139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:36.849167 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:36.852466 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.853292 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.853296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.853299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.853511 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.853902 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.853905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.853907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.854285 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:36.854570 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:36.855125 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 14:01:36.855399 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 14:01:36.857153 13163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07: Bootstrap starting.
I20260927 14:01:36.860226 22635 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:01:36.860942 13163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:36.861981 13163 log.cc:824] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:36.864583 13163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07: No bootstrap required, opened a new log
I20260927 14:01:36.862522 12990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fdc84d3e979a41a6805f38e94db607f0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51084
I20260927 14:01:36.871543 13163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } }
I20260927 14:01:36.871876 13163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318369e0eabf4857b70f375e94638d07, State: Initialized, Role: FOLLOWER
I20260927 14:01:36.872295 13163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } }
I20260927 14:01:36.873200 13169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } } }
I20260927 14:01:36.873661 13169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:36.874135 13163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 14:01:36.883222 13180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:01:36.883363 13180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:01:36.880951 13096 master_runner.cc:186] Error getting master registration for 127.22.26.254:37917: OK, instance_id { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" instance_seqno: 1790517695782337 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 14:01:36.892215 13096 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 14:01:36.923439 13015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } attempt: 4
Sep 27 14:01:36 dist-test-slave-gkks krb5kdc[12935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517695, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.254@KRBTEST.COM for kudu/127.22.26.252@KRBTEST.COM
I20260927 14:01:36.939718 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.924311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60200)
0927 14:01:36.924575 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:36.924593 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:36.924717 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:36.925275 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:36.925281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:36.925825 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:36.926052 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.926066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.927193 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.927200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.928695 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:36.928713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.929906 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.929914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:36.930191 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.930939 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:36.930963 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:36.933860 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:36.937118 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.937124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.937131 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.937510 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.937901 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:36.937907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:36.937910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:36.938057 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:36.938662 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:36.938667 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:36.938843 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 14:01:36.938983 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 14:01:36.939286 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260927 14:01:36.939715 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:36.924323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.22.26.252:37853)
0927 14:01:36.924574 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:36.924581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:36.924656 (+ 75us) server_negotiation.cc:407] Connection header received
0927 14:01:36.924846 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:36.924853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:36.924945 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:36.925067 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:36.926218 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.926994 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.928874 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:36.929684 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:36.929807 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:36.934041 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:36.934047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:36.934053 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:36.934091 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:36.936969 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.937640 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.937648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.937657 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.937777 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:36.938184 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:36.938189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:36.938192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:36.938534 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:36.938721 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:36.939114 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 14:01:36.939334 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 14:01:36.945478 13015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0: Bootstrap starting.
I20260927 14:01:36.949597 13015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:36.950886 13015 log.cc:824] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:36.955783 13015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0: No bootstrap required, opened a new log
I20260927 14:01:36.963404 13015 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } }
I20260927 14:01:36.963800 13015 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc84d3e979a41a6805f38e94db607f0, State: Initialized, Role: FOLLOWER
I20260927 14:01:36.964313 13015 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } }
I20260927 14:01:36.965569 13183 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } } }
I20260927 14:01:36.966120 13183 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:36.966744 13015 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:36.966861 12990 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 14:01:36.967038 12990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:36.971256 12990 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } }
I20260927 14:01:36.974553 12990 leader_election.cc:288] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [CANDIDATE]: Term 1 election: Requested vote from peers 0f304acaaea8481eae31fc4781565ef0 (127.22.26.253:43051), 318369e0eabf4857b70f375e94638d07 (127.22.26.252:37853)
I20260927 14:01:36.974514 13138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdc84d3e979a41a6805f38e94db607f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "318369e0eabf4857b70f375e94638d07"
I20260927 14:01:36.975138 13138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:36.979568 13138 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc84d3e979a41a6805f38e94db607f0 in term 1.
I20260927 14:01:36.980343 12965 leader_election.cc:302] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 318369e0eabf4857b70f375e94638d07, fdc84d3e979a41a6805f38e94db607f0; no voters:
I20260927 14:01:36.980937 13183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:01:36.984997 13183 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [term 1 LEADER]: Becoming Leader. State: Replica: fdc84d3e979a41a6805f38e94db607f0, State: Running, Role: LEADER
I20260927 14:01:36.985728 13183 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } }
I20260927 14:01:36.992203 13195 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdc84d3e979a41a6805f38e94db607f0. Latest consensus state: current_term: 1 leader_uuid: "fdc84d3e979a41a6805f38e94db607f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } } }
I20260927 14:01:36.992575 13195 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:36.993736 13199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:01:37.003031 13089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } attempt: 5
W20260927 14:01:37.020327 13117 tablet.cc:2369] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:01:37.021917 12969 tablet.cc:2369] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 14:01:37 dist-test-slave-gkks krb5kdc[12935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517696, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.26.253@KRBTEST.COM for kudu/127.22.26.252@KRBTEST.COM
I20260927 14:01:37.054334 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.003555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.26.252:37853 (local address 127.0.0.1:60214)
0927 14:01:37.003911 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 14:01:37.003931 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 14:01:37.004035 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:01:37.007445 (+ 3410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:01:37.007454 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:01:37.008086 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:01:37.008387 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:37.008406 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.014357 (+ 5951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.014368 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:37.016260 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:01:37.016297 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.023715 (+ 7418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.023726 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:01:37.024105 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.026480 (+ 2375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:01:37.026513 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:01:37.030469 (+ 3956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:01:37.046263 (+ 15794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:37.046272 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:37.046278 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:37.047074 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:37.050266 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:01:37.050275 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:01:37.050282 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:01:37.050518 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:01:37.052956 (+ 2438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:01:37.052963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:01:37.053176 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 14:01:37.053414 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 14:01:37.053836 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260927 14:01:37.002354 12948 master_runner.cc:186] Error getting master registration for 127.22.26.253:43051: OK, instance_id { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" instance_seqno: 1790517696278388 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 14:01:37.062469 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.004252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.22.26.252:37853)
0927 14:01:37.006718 (+ 2466us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.006726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.006752 (+ 26us) server_negotiation.cc:407] Connection header received
0927 14:01:37.006827 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.006836 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.006931 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.007047 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.009616 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.011036 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.016911 (+ 5875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.017890 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.018050 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.031629 (+ 13579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:37.031641 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:37.031650 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:37.031709 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:37.042770 (+ 11061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:37.049396 (+ 6626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:37.049407 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:37.049413 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:37.049649 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:37.051857 (+ 2208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:37.051866 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:37.051872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:37.052376 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:37.052567 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.061569 (+ 9002us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.061919 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2323}
I20260927 14:01:37.083324 12948 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 14:01:37.091068 13199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:01:37.101188 13089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0: Bootstrap starting.
I20260927 14:01:37.105417 13089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0: Neither blocks nor log segments found. Creating new log.
I20260927 14:01:37.106802 13089 log.cc:824] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0: Log is configured to *not* fsync() on all Append() calls
I20260927 14:01:37.107050 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.009186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.22.26.253:43051)
0927 14:01:37.009387 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.009396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.070877 (+ 61481us) server_negotiation.cc:407] Connection header received
0927 14:01:37.071177 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.071186 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.071296 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.071379 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.087248 (+ 15869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.088415 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.091704 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.092390 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.092494 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.098090 (+ 5596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:37.098098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:37.098104 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:37.098151 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:37.102167 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:37.103705 (+ 1538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:37.103711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:37.103715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:37.103888 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:37.104637 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:37.104642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:37.104645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:37.105033 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:37.105273 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.106301 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.106556 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 14:01:37.117950 13089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0: No bootstrap required, opened a new log
I20260927 14:01:37.119982 13138 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07 [term 1 FOLLOWER]: Refusing update from remote peer fdc84d3e979a41a6805f38e94db607f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:01:37.121034 13195 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:01:37.127233 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.006361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.22.26.252:37853)
0927 14:01:37.063133 (+ 56772us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.063143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.074989 (+ 11846us) server_negotiation.cc:407] Connection header received
0927 14:01:37.075069 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.075096 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.075212 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.075341 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.078064 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.079566 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.097275 (+ 17709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.098010 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.098140 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.113791 (+ 15651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:37.113801 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:37.113809 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:37.113863 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:37.117529 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:37.118395 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:37.118401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:37.118404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:37.118550 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:37.124035 (+ 5485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:37.124041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:37.124044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:37.124450 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:37.124721 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.126336 (+ 1615us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.126674 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56562,"thread_start_us":88923,"threads_started":1}
I20260927 14:01:37.133345 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.000498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.22.26.254:37917)
0927 14:01:37.000881 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.000888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.066954 (+ 66066us) server_negotiation.cc:407] Connection header received
0927 14:01:37.067349 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.067356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.067469 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.067552 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.087344 (+ 19792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.088859 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.104661 (+ 15802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.105389 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.105499 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.118753 (+ 13254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:01:37.118763 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:01:37.118769 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:01:37.118842 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:01:37.122886 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:37.124819 (+ 1933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:37.124824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:37.124828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:37.124996 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:01:37.130950 (+ 5954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:01:37.130959 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:01:37.130965 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:01:37.131462 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:01:37.131769 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.132537 (+ 768us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.132837 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260927 14:01:37.136097 13089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } }
I20260927 14:01:37.136847 13089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f304acaaea8481eae31fc4781565ef0, State: Initialized, Role: FOLLOWER
I20260927 14:01:37.137732 13089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } }
I20260927 14:01:37.144588 13169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdc84d3e979a41a6805f38e94db607f0. Latest consensus state: current_term: 1 leader_uuid: "fdc84d3e979a41a6805f38e94db607f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } } }
I20260927 14:01:37.144909 13169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:37.147688 13208 mvcc.cc:205] Tried to move back new op lower bound from 7333960487390638080 to 7333960486860517376. Current Snapshot: MvccSnapshot[applied={T|T < 7333960487390638080}]
I20260927 14:01:37.150106 13169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdc84d3e979a41a6805f38e94db607f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } } }
I20260927 14:01:37.150420 13169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:37.156706 13064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdc84d3e979a41a6805f38e94db607f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0f304acaaea8481eae31fc4781565ef0"
I20260927 14:01:37.157069 13063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:01:37.157567 13089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:01:37.160914 13063 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0 [term 1 FOLLOWER]: Refusing update from remote peer fdc84d3e979a41a6805f38e94db607f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:01:37.161913 13195 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:01:37.154410 13211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } } }
I20260927 14:01:37.163939 13211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:37.164263 13064 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc84d3e979a41a6805f38e94db607f0 in term 1.
I20260927 14:01:37.169032 13195 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdc84d3e979a41a6805f38e94db607f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } } }
I20260927 14:01:37.169310 13195 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:37.170161 13199 catalog_manager.cc:1347] Generated new cluster ID: 384f4608b15243ddbc3bab5d5ff98d3e
I20260927 14:01:37.170258 13199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:01:37.195034 13199 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:01:37.196941 13199 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:01:37.202802 13211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdc84d3e979a41a6805f38e94db607f0. Latest consensus state: current_term: 1 leader_uuid: "fdc84d3e979a41a6805f38e94db607f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } } }
I20260927 14:01:37.203208 13211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:37.203346 13195 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdc84d3e979a41a6805f38e94db607f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } } }
I20260927 14:01:37.203593 13195 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0 [sys.catalog]: This master's current role is: LEADER
I20260927 14:01:37.205780 13217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdc84d3e979a41a6805f38e94db607f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc84d3e979a41a6805f38e94db607f0" member_type: VOTER last_known_addr { host: "127.22.26.254" port: 37917 } } peers { permanent_uuid: "0f304acaaea8481eae31fc4781565ef0" member_type: VOTER last_known_addr { host: "127.22.26.253" port: 43051 } } peers { permanent_uuid: "318369e0eabf4857b70f375e94638d07" member_type: VOTER last_known_addr { host: "127.22.26.252" port: 37853 } } }
I20260927 14:01:37.206113 13217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:01:37.235759 13199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdc84d3e979a41a6805f38e94db607f0: Generated new TSK 0
I20260927 14:01:37.236979 13199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:01:37.250438 13230 catalog_manager.cc:1259] Loaded cluster ID: 384f4608b15243ddbc3bab5d5ff98d3e
I20260927 14:01:37.250627 13230 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0: loading cluster ID for follower catalog manager: success
I20260927 14:01:37.254585 13230 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0: acquiring CA information for follower catalog manager: success
I20260927 14:01:37.257056 13230 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0f304acaaea8481eae31fc4781565ef0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 14:01:37.375447 13018 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:37.354472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.22.26.254:37917)
0927 14:01:37.354738 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.354745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.372733 (+ 17988us) server_negotiation.cc:407] Connection header received
0927 14:01:37.373146 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.373156 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.373290 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.373385 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.375045 (+ 1660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51094: BlockingRecv error: recv got EOF from 127.0.0.1:51094 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260927 14:01:37.378374 13093 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:37.356446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.22.26.253:43051)
0927 14:01:37.356674 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.356681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.376308 (+ 19627us) server_negotiation.cc:407] Connection header received
0927 14:01:37.376620 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.376629 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.376736 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.376818 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.378071 (+ 1253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38186: BlockingRecv error: recv got EOF from 127.0.0.1:38186 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 14:01:37.380434 13166 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:37.366247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.22.26.252:37853)
0927 14:01:37.366424 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.366430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.378888 (+ 12458us) server_negotiation.cc:407] Connection header received
0927 14:01:37.378988 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.378994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.379111 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.379215 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.380192 (+ 977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60228: BlockingRecv error: recv got EOF from 127.0.0.1:60228 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:01:37.430958 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.411819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.22.26.253:43051)
0927 14:01:37.412017 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.412025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.422603 (+ 10578us) server_negotiation.cc:407] Connection header received
0927 14:01:37.423222 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.423232 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.423477 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.423635 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:37.424350 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.425547 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.427717 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.428422 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.428529 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.429363 (+ 834us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:37.430074 (+ 711us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:37.430211 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.430466 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.430568 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 14:01:37.477680 13166 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:37.464388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.22.26.252:37853)
0927 14:01:37.464645 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.464653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.475287 (+ 10634us) server_negotiation.cc:407] Connection header received
0927 14:01:37.475919 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.475926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.476079 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.476173 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.477376 (+ 1203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60236: BlockingRecv error: recv got EOF from 127.0.0.1:60236 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 14:01:37.532582 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.513276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.22.26.253:43051)
0927 14:01:37.513489 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.513496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.524966 (+ 11470us) server_negotiation.cc:407] Connection header received
0927 14:01:37.525516 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.525523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.525665 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.525733 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:37.526622 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.527916 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.530103 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.530823 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.530929 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.531631 (+ 702us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:37.531809 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:37.531880 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.532139 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.532237 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 14:01:37.586642 13166 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:37.573518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.22.26.252:37853)
0927 14:01:37.573699 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.573705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.584474 (+ 10769us) server_negotiation.cc:407] Connection header received
0927 14:01:37.585015 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.585023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.585142 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.585230 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.586385 (+ 1155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60252: BlockingRecv error: recv got EOF from 127.0.0.1:60252 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:01:37.640966 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.623216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.22.26.253:43051)
0927 14:01:37.623420 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.623431 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.634115 (+ 10684us) server_negotiation.cc:407] Connection header received
0927 14:01:37.634665 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.634671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.634758 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.634823 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:37.635536 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.636678 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.638665 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.639311 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.639416 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.640098 (+ 682us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:37.640217 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:37.640268 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.640481 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.640568 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 14:01:37.649679 13166 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:37.647440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.22.26.252:37853)
0927 14:01:37.647691 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.647700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.648222 (+ 522us) server_negotiation.cc:407] Connection header received
0927 14:01:37.648431 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.648437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.648588 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.648681 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.649435 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60256: BlockingRecv error: recv got EOF from 127.0.0.1:60256 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 14:01:37.663158 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.657188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.22.26.253:43051)
0927 14:01:37.657488 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.657497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.658060 (+ 563us) server_negotiation.cc:407] Connection header received
0927 14:01:37.658310 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.658316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.658461 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.658567 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:37.659067 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.659864 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.661294 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.661777 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.661926 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.662327 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:37.662438 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:37.662544 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.662699 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.662792 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
W20260927 14:01:37.674887 13166 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:37.672999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.22.26.252:37853)
0927 14:01:37.673273 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.673280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.673376 (+ 96us) server_negotiation.cc:407] Connection header received
0927 14:01:37.673636 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.673642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.673728 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.673839 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.674697 (+ 858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60258: BlockingRecv error: recv got EOF from 127.0.0.1:60258 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
I20260927 14:01:37.694866 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.689073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.22.26.253:43051)
0927 14:01:37.689330 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.689340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.689928 (+ 588us) server_negotiation.cc:407] Connection header received
0927 14:01:37.690161 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.690166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.690238 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.690410 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:37.690820 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.691651 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.693003 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.693454 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.693599 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.694047 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:37.694171 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:37.694280 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.694433 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.694513 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 14:01:37.705215 13166 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:37.701066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.22.26.252:37853)
0927 14:01:37.701307 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.701316 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.703794 (+ 2478us) server_negotiation.cc:407] Connection header received
0927 14:01:37.704010 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.704014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.704087 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.704194 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.704953 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60268: BlockingRecv error: recv got EOF from 127.0.0.1:60268 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 14:01:37.730428 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.724345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.22.26.253:43051)
0927 14:01:37.724607 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.724615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.725159 (+ 544us) server_negotiation.cc:407] Connection header received
0927 14:01:37.725403 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.725408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.725474 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.725577 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:37.726025 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.726832 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.728355 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.728811 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.728954 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.729514 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:37.729685 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:37.729756 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.730021 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.730130 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260927 14:01:37.741381 13166 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:37.738632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.22.26.252:37853)
0927 14:01:37.738822 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.738827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.739484 (+ 657us) server_negotiation.cc:407] Connection header received
0927 14:01:37.739905 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.739912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.740021 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.740128 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.741145 (+ 1017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60284: BlockingRecv error: recv got EOF from 127.0.0.1:60284 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 14:01:37.784143 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.777580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.22.26.253:43051)
0927 14:01:37.777919 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.777925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.778229 (+ 304us) server_negotiation.cc:407] Connection header received
0927 14:01:37.778489 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.778494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.778587 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.778700 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:37.779274 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.780111 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.782008 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.782482 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.782584 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.783194 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:37.783304 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:37.783384 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.783660 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.783730 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 14:01:37.819336 13166 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:01:37.816366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60296 (local address 127.22.26.252:37853)
0927 14:01:37.816594 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.816616 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.817915 (+ 1299us) server_negotiation.cc:407] Connection header received
0927 14:01:37.818060 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.818066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.818172 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.818290 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:01:37.819123 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60296: BlockingRecv error: recv got EOF from 127.0.0.1:60296 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 14:01:37.886034 13180 catalog_manager.cc:1259] Loaded cluster ID: 384f4608b15243ddbc3bab5d5ff98d3e
I20260927 14:01:37.886152 13180 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07: loading cluster ID for follower catalog manager: success
I20260927 14:01:37.889844 13180 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07: acquiring CA information for follower catalog manager: success
I20260927 14:01:37.892215 13180 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 318369e0eabf4857b70f375e94638d07: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 14:01:37.893308 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.887485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.22.26.253:43051)
0927 14:01:37.887711 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.887717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.888072 (+ 355us) server_negotiation.cc:407] Connection header received
0927 14:01:37.888315 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.888319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.888402 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.888476 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:37.888995 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.889720 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.891678 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.892071 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.892162 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.892632 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:37.892729 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:37.892801 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.892940 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.893004 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 14:01:37.903965 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.898689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.22.26.252:37853)
0927 14:01:37.898947 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.898953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.899261 (+ 308us) server_negotiation.cc:407] Connection header received
0927 14:01:37.899401 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.899405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.899556 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.899652 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:37.900006 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.900676 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.901895 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.902299 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.902398 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.902796 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:37.903381 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:37.903507 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.903577 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.903640 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 14:01:37.929445 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.909711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.22.26.254:37917)
0927 14:01:37.922088 (+ 12377us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.922098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.922260 (+ 162us) server_negotiation.cc:407] Connection header received
0927 14:01:37.922563 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.922568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.922832 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.923031 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:37.923915 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.924864 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.926945 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.927622 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.927739 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.928330 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:37.928761 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:37.928887 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.929019 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.929114 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12174,"thread_start_us":12939,"threads_started":1}
I20260927 14:01:37.929507 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.910168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.22.26.253:43051)
0927 14:01:37.910363 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.910368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.922900 (+ 12532us) server_negotiation.cc:407] Connection header received
0927 14:01:37.923208 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.923214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.923310 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.923406 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:37.924238 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.925330 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.927471 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.928044 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.928186 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.928762 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:37.928914 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:37.929000 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.929193 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.929275 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 14:01:37.929608 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:01:37.910246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.22.26.252:37853)
0927 14:01:37.910397 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 14:01:37.910406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:01:37.923343 (+ 12937us) server_negotiation.cc:407] Connection header received
0927 14:01:37.923549 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:01:37.923555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:01:37.923654 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:01:37.923751 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:01:37.924478 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.925500 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.927499 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:01:37.928062 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:01:37.928176 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:01:37.928818 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:01:37.928968 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:01:37.929046 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 14:01:37.929244 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 14:01:37.929341 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:01:37.934674 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 12948
I20260927 14:01:37.935739 13010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:01:38.048391 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 12948
I20260927 14:01:38.070363 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 13022
I20260927 14:01:38.071589 13084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:01:38.181593 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 13022
I20260927 14:01:38.201781 22635 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu pid 13096
I20260927 14:01:38.202963 13158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 14:01:38.311297 22635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUeHuEC/build/asan/bin/kudu with pid 13096
2026-09-27T14:01:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4345 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9436 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (367466 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST